{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ML2021Spring - HW1.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz0_QVkxCrX3"
      },
      "source": [
        "# **Homework 1: COVID-19 Cases Prediction (Regression)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeZnPAiwDRWG"
      },
      "source": [
        "Author: Heng-Jui Chang\n",
        "\n",
        "Slides: https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.pdf  \n",
        "Videos (Mandarin): https://cool.ntu.edu.tw/courses/4793/modules/items/172854  \n",
        "https://cool.ntu.edu.tw/courses/4793/modules/items/172853  \n",
        "Video (English): https://cool.ntu.edu.tw/courses/4793/modules/items/176529\n",
        "\n",
        "\n",
        "Objectives:\n",
        "* Solve a regression problem with deep neural networks (DNN).\n",
        "* Understand basic DNN training tips.\n",
        "* Get familiar with PyTorch.\n",
        "\n",
        "If any questions, please contact the TAs via TA hours, NTU COOL, or email.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jx3x1nDkG-Uy"
      },
      "source": [
        "# **Download Data**\n",
        "\n",
        "\n",
        "If the Google drive links are dead, you can download data from [kaggle](https://www.kaggle.com/c/ml2021spring-hw1/data), and upload data manually to the workspace."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMj55YDKG6ch",
        "outputId": "c8847fda-46b3-43dd-a525-c3b080254901"
      },
      "source": [
        "tr_path = 'covid.train.csv'  # path to training data\n",
        "tt_path = 'covid.test.csv'   # path to testing data\n",
        "\n",
        "!gdown --id '19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF' --output covid.train.csv\n",
        "!gdown --id '1CE240jLm2npU-tdz81-oVKEF3T2yfT1O' --output covid.test.csv"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF\n",
            "To: /content/covid.train.csv\n",
            "100% 2.00M/2.00M [00:00<00:00, 133MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1CE240jLm2npU-tdz81-oVKEF3T2yfT1O\n",
            "To: /content/covid.test.csv\n",
            "100% 651k/651k [00:00<00:00, 93.7MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wS_4-77xHk44"
      },
      "source": [
        "# **Import Some Packages**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-onQd4JNA5H"
      },
      "source": [
        "# PyTorch\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "# For data preprocess\n",
        "import numpy as np\n",
        "import csv\n",
        "import os\n",
        "\n",
        "# For plotting\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.pyplot import figure\n",
        "\n",
        "myseed = 42069  # set a random seed for reproducibility\n",
        "torch.backends.cudnn.deterministic = True\n",
        "torch.backends.cudnn.benchmark = False\n",
        "np.random.seed(myseed)\n",
        "torch.manual_seed(myseed)\n",
        "if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed_all(myseed)"
      ],
      "execution_count": 95,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BtE3b6JEH7rw"
      },
      "source": [
        "# **Some Utilities**\n",
        "\n",
        "You do not need to modify this part."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FWMT3uf1NGQp"
      },
      "source": [
        "def get_device():\n",
        "    ''' Get device (if GPU is available, use GPU) '''\n",
        "    return 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "def plot_learning_curve(loss_record, title=''):\n",
        "    ''' Plot learning curve of your DNN (train & dev loss) '''\n",
        "    total_steps = len(loss_record['train'])\n",
        "    x_1 = range(total_steps)\n",
        "    x_2 = x_1[::len(loss_record['train']) // len(loss_record['dev'])]\n",
        "    figure(figsize=(6, 4))\n",
        "    plt.plot(x_1, loss_record['train'], c='tab:red', label='train')\n",
        "    plt.plot(x_2, loss_record['dev'], c='tab:cyan', label='dev')\n",
        "    plt.ylim(0.0, 5.)\n",
        "    plt.xlabel('Training steps')\n",
        "    plt.ylabel('MSE loss')\n",
        "    plt.title('Learning curve of {}'.format(title))\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "def plot_pred(dv_set, model, device, lim=35., preds=None, targets=None):\n",
        "    ''' Plot prediction of your DNN '''\n",
        "    if preds is None or targets is None:\n",
        "        model.eval()\n",
        "        preds, targets = [], []\n",
        "        for x, y in dv_set:\n",
        "            x, y = x.to(device), y.to(device)\n",
        "            with torch.no_grad():\n",
        "                pred = model(x)\n",
        "                preds.append(pred.detach().cpu())\n",
        "                targets.append(y.detach().cpu())\n",
        "        preds = torch.cat(preds, dim=0).numpy()\n",
        "        targets = torch.cat(targets, dim=0).numpy()\n",
        "\n",
        "    figure(figsize=(5, 5))\n",
        "    plt.scatter(targets, preds, c='r', alpha=0.5)\n",
        "    plt.plot([-0.2, lim], [-0.2, lim], c='b')\n",
        "    plt.xlim(-0.2, lim)\n",
        "    plt.ylim(-0.2, lim)\n",
        "    plt.xlabel('ground truth value')\n",
        "    plt.ylabel('predicted value')\n",
        "    plt.title('Ground Truth v.s. Prediction')\n",
        "    plt.show()"
      ],
      "execution_count": 96,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39U_XFX6KOoj"
      },
      "source": [
        "# **Preprocess**\n",
        "\n",
        "We have three kinds of datasets:\n",
        "* `train`: for training\n",
        "* `dev`: for validation\n",
        "* `test`: for testing (w/o target value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TQ-MdwpLL7Dt"
      },
      "source": [
        "## **Dataset**\n",
        "\n",
        "The `COVID19Dataset` below does:\n",
        "* read `.csv` files\n",
        "* extract features\n",
        "* split `covid.train.csv` into train/dev sets\n",
        "* normalize features\n",
        "\n",
        "Finishing `TODO` below might make you pass medium baseline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zlpIp9ANJRU"
      },
      "source": [
        "class COVID19Dataset(Dataset):\n",
        "    ''' Dataset for loading and preprocessing the COVID19 dataset '''\n",
        "    def __init__(self,\n",
        "                 path,\n",
        "                 mode='train',\n",
        "                 target_only=False):\n",
        "        self.mode = mode\n",
        "\n",
        "        # Read data into numpy arrays\n",
        "        with open(path, 'r') as fp:\n",
        "            data = list(csv.reader(fp))\n",
        "            data = np.array(data[1:])[:, 1:].astype(float)\n",
        "        \n",
        "        if not target_only:\n",
        "            feats = list(range(93))\n",
        "        else:\n",
        "            # TODO: Using 40 states & 2 tested_positive features (indices = 57 & 75)\n",
        "            feats = list(range(40)) + list(range(40,44)) + [57] + list(range(58,62)) + [75] + list(range(76,80))\n",
        "\n",
        "        if mode == 'test':\n",
        "            # Testing data\n",
        "            # data: 893 x 93 (40 states + day 1 (18) + day 2 (18) + day 3 (17))\n",
        "            data = data[:, feats]\n",
        "            self.data = torch.FloatTensor(data)\n",
        "        else:\n",
        "            # Training data (train/dev sets)\n",
        "            # data: 2700 x 94 (40 states + day 1 (18) + day 2 (18) + day 3 (18))\n",
        "            target = data[:, -1]\n",
        "            data = data[:, feats]\n",
        "            \n",
        "            # Splitting training data into train & dev sets\n",
        "            if mode == 'train':\n",
        "                indices = [i for i in range(len(data)) if i % 10 != 0]\n",
        "            elif mode == 'dev':\n",
        "                indices = [i for i in range(len(data)) if i % 10 == 0]\n",
        "            \n",
        "            # Convert data into PyTorch tensors\n",
        "            self.data = torch.FloatTensor(data[indices])\n",
        "            self.target = torch.FloatTensor(target[indices])\n",
        "\n",
        "        # Normalize features (you may remove this part to see what will happen)\n",
        "        self.data[:, 40:] = \\\n",
        "            (self.data[:, 40:] - self.data[:, 40:].mean(dim=0, keepdim=True)) \\\n",
        "            / self.data[:, 40:].std(dim=0, keepdim=True)\n",
        "\n",
        "        self.dim = self.data.shape[1]\n",
        "\n",
        "        print('Finished reading the {} set of COVID19 Dataset ({} samples found, each dim = {})'\n",
        "              .format(mode, len(self.data), self.dim))\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        # Returns one sample at a time\n",
        "        if self.mode in ['train', 'dev']:\n",
        "            # For training\n",
        "            return self.data[index], self.target[index]\n",
        "        else:\n",
        "            # For testing (no target)\n",
        "            return self.data[index]\n",
        "\n",
        "    def __len__(self):\n",
        "        # Returns the size of the dataset\n",
        "        return len(self.data)"
      ],
      "execution_count": 97,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlhTlkE7MDo3"
      },
      "source": [
        "## **DataLoader**\n",
        "\n",
        "A `DataLoader` loads data from a given `Dataset` into batches.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hlhLk5t6MBX3"
      },
      "source": [
        "def prep_dataloader(path, mode, batch_size, n_jobs=0, target_only=False):\n",
        "    ''' Generates a dataset, then is put into a dataloader. '''\n",
        "    dataset = COVID19Dataset(path, mode=mode, target_only=target_only)  # Construct dataset\n",
        "    dataloader = DataLoader(\n",
        "        dataset, batch_size,\n",
        "        shuffle=(mode == 'train'), drop_last=False,\n",
        "        num_workers=n_jobs, pin_memory=True)                            # Construct dataloader\n",
        "    return dataloader"
      ],
      "execution_count": 98,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGuycwR0MeQB"
      },
      "source": [
        "# **Deep Neural Network**\n",
        "\n",
        "`NeuralNet` is an `nn.Module` designed for regression.\n",
        "The DNN consists of 2 fully-connected layers with ReLU activation.\n",
        "This module also included a function `cal_loss` for calculating loss.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49-uXYovOAI0"
      },
      "source": [
        "class NeuralNet(nn.Module):\n",
        "    ''' A simple fully-connected deep neural network '''\n",
        "    def __init__(self, input_dim):\n",
        "        super(NeuralNet, self).__init__()\n",
        "\n",
        "        # Define your neural network here\n",
        "        # TODO: How to modify this model to achieve better performance?\n",
        "        self.net = nn.Sequential(\n",
        "            nn.Linear(input_dim, 64),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(64, 1)\n",
        "        )\n",
        "\n",
        "        # Mean squared error loss\n",
        "        self.criterion = nn.MSELoss(reduction='mean')\n",
        "\n",
        "    def forward(self, x):\n",
        "        ''' Given input of size (batch_size x input_dim), compute output of the network '''\n",
        "        return self.net(x).squeeze(1)\n",
        "\n",
        "    def cal_loss(self, pred, target):\n",
        "        ''' Calculate loss '''\n",
        "        # TODO: you may implement L1/L2 regularization here\n",
        "        return torch.sqrt(self.criterion(pred, target))"
      ],
      "execution_count": 99,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DvFWVjZ5Nvga"
      },
      "source": [
        "# **Train/Dev/Test**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAM8QecJOyqn"
      },
      "source": [
        "## **Training**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOqcmYzMO7jB"
      },
      "source": [
        "def train(tr_set, dv_set, model, config, device):\n",
        "    ''' DNN training '''\n",
        "\n",
        "    n_epochs = config['n_epochs']  # Maximum number of epochs\n",
        "\n",
        "    # Setup optimizer\n",
        "    optimizer = getattr(torch.optim, config['optimizer'])(\n",
        "        model.parameters(), **config['optim_hparas'])\n",
        "\n",
        "    min_mse = 1000.\n",
        "    loss_record = {'train': [], 'dev': []}      # for recording training loss\n",
        "    early_stop_cnt = 0\n",
        "    epoch = 0\n",
        "    while epoch < n_epochs:\n",
        "        model.train()                           # set model to training mode\n",
        "        for x, y in tr_set:                     # iterate through the dataloader\n",
        "            optimizer.zero_grad()               # set gradient to zero\n",
        "            x, y = x.to(device), y.to(device)   # move data to device (cpu/cuda)\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "            mse_loss.backward()                 # compute gradient (backpropagation)\n",
        "            optimizer.step()                    # update model with optimizer\n",
        "            loss_record['train'].append(mse_loss.detach().cpu().item())\n",
        "\n",
        "        # After each epoch, test your model on the validation (development) set.\n",
        "        dev_mse = dev(dv_set, model, device)\n",
        "        if dev_mse < min_mse:\n",
        "            # Save model if your model improved\n",
        "            min_mse = dev_mse\n",
        "            print('Saving model (epoch = {:4d}, loss = {:.4f})'\n",
        "                .format(epoch + 1, min_mse))\n",
        "            torch.save(model.state_dict(), config['save_path'])  # Save model to specified path\n",
        "            early_stop_cnt = 0\n",
        "        else:\n",
        "            early_stop_cnt += 1\n",
        "\n",
        "        epoch += 1\n",
        "        loss_record['dev'].append(dev_mse)\n",
        "        if early_stop_cnt > config['early_stop']:\n",
        "            # Stop training if your model stops improving for \"config['early_stop']\" epochs.\n",
        "            break\n",
        "\n",
        "    print('Finished training after {} epochs'.format(epoch))\n",
        "    return min_mse, loss_record"
      ],
      "execution_count": 100,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0hSd4Bn3O2PL"
      },
      "source": [
        "## **Validation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yrxrD3YsN3U2"
      },
      "source": [
        "def dev(dv_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    total_loss = 0\n",
        "    for x, y in dv_set:                         # iterate through the dataloader\n",
        "        x, y = x.to(device), y.to(device)       # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "        total_loss += mse_loss.detach().cpu().item() * len(x)  # accumulate loss\n",
        "    total_loss = total_loss / len(dv_set.dataset)              # compute averaged loss\n",
        "\n",
        "    return total_loss"
      ],
      "execution_count": 101,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0pdrhQAO41L"
      },
      "source": [
        "## **Testing**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSBMRFlYN5tB"
      },
      "source": [
        "def test(tt_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    preds = []\n",
        "    for x in tt_set:                            # iterate through the dataloader\n",
        "        x = x.to(device)                        # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            preds.append(pred.detach().cpu())   # collect prediction\n",
        "    preds = torch.cat(preds, dim=0).numpy()     # concatenate all predictions and convert to a numpy array\n",
        "    return preds"
      ],
      "execution_count": 102,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvckkF5dvf0j"
      },
      "source": [
        "# **Setup Hyper-parameters**\n",
        "\n",
        "`config` contains hyper-parameters for training and the path to save your model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NPXpdumwPjE7"
      },
      "source": [
        "device = get_device()                 # get the current available device ('cpu' or 'cuda')\n",
        "os.makedirs('models', exist_ok=True)  # The trained model will be saved to ./models/\n",
        "target_only = True                   # TODO: Using 40 states & 2 tested_positive features\n",
        "\n",
        "# TODO: How to tune these hyper-parameters to improve your model's performance?\n",
        "config = {\n",
        "    'n_epochs': 3000,                # maximum number of epochs\n",
        "    'batch_size': 64,               # mini-batch size for dataloader\n",
        "    'optimizer': 'SGD',              # optimization algorithm (optimizer in torch.optim)\n",
        "    'optim_hparas': {                # hyper-parameters for the optimizer (depends on which optimizer you are using)\n",
        "        'lr': 0.001,                 # learning rate of SGD\n",
        "        'momentum': 0.9,              # momentum for SGD\n",
        "        'weight_decay': 1e-5\n",
        "    },\n",
        "    'early_stop': 200,               # early stopping epochs (the number epochs since your model's last improvement)\n",
        "    'save_path': 'models/model.pth'  # your model will be saved here\n",
        "}"
      ],
      "execution_count": 103,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6j1eOV3TOH-j"
      },
      "source": [
        "# **Load data and model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eNrYBMmePLKm",
        "outputId": "f2c1f00a-014e-4744-fec6-840cecc56e46"
      },
      "source": [
        "tr_set = prep_dataloader(tr_path, 'train', config['batch_size'], target_only=target_only)\n",
        "dv_set = prep_dataloader(tr_path, 'dev', config['batch_size'], target_only=target_only)\n",
        "tt_set = prep_dataloader(tt_path, 'test', config['batch_size'], target_only=target_only)"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Finished reading the train set of COVID19 Dataset (2430 samples found, each dim = 54)\n",
            "Finished reading the dev set of COVID19 Dataset (270 samples found, each dim = 54)\n",
            "Finished reading the test set of COVID19 Dataset (893 samples found, each dim = 54)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FHylSirLP9oh"
      },
      "source": [
        "model = NeuralNet(tr_set.dataset.dim).to(device)  # Construct model and move to device"
      ],
      "execution_count": 105,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sX2B_zgSOPTJ"
      },
      "source": [
        "# **Start Training!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GrEbUxazQAAZ",
        "outputId": "60d91fc4-a42a-4d64-9c9c-5404ae6b8bbf"
      },
      "source": [
        "model_loss, model_loss_record = train(tr_set, dv_set, model, config, device)"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving model (epoch =    1, loss = 16.8509)\n",
            "Saving model (epoch =    2, loss = 14.7211)\n",
            "Saving model (epoch =    3, loss = 10.3639)\n",
            "Saving model (epoch =    4, loss = 8.8282)\n",
            "Saving model (epoch =    5, loss = 7.8176)\n",
            "Saving model (epoch =    6, loss = 6.8041)\n",
            "Saving model (epoch =    7, loss = 5.8081)\n",
            "Saving model (epoch =    8, loss = 4.3797)\n",
            "Saving model (epoch =    9, loss = 2.5636)\n",
            "Saving model (epoch =   10, loss = 1.4485)\n",
            "Saving model (epoch =   11, loss = 1.1065)\n",
            "Saving model (epoch =   12, loss = 1.0548)\n",
            "Saving model (epoch =   13, loss = 1.0298)\n",
            "Saving model (epoch =   15, loss = 1.0001)\n",
            "Saving model (epoch =   16, loss = 0.9936)\n",
            "Saving model (epoch =   19, loss = 0.9684)\n",
            "Saving model (epoch =   22, loss = 0.9505)\n",
            "Saving model (epoch =   24, loss = 0.9406)\n",
            "Saving model (epoch =   25, loss = 0.9404)\n",
            "Saving model (epoch =   26, loss = 0.9378)\n",
            "Saving model (epoch =   28, loss = 0.9316)\n",
            "Saving model (epoch =   31, loss = 0.9221)\n",
            "Saving model (epoch =   36, loss = 0.9193)\n",
            "Saving model (epoch =   39, loss = 0.9180)\n",
            "Saving model (epoch =   41, loss = 0.9171)\n",
            "Saving model (epoch =   42, loss = 0.9103)\n",
            "Saving model (epoch =   47, loss = 0.9092)\n",
            "Saving model (epoch =   53, loss = 0.9068)\n",
            "Saving model (epoch =   58, loss = 0.9047)\n",
            "Saving model (epoch =   69, loss = 0.9015)\n",
            "Saving model (epoch =   97, loss = 0.8998)\n",
            "Saving model (epoch =  114, loss = 0.8978)\n",
            "Saving model (epoch =  251, loss = 0.8964)\n",
            "Saving model (epoch =  255, loss = 0.8963)\n",
            "Saving model (epoch =  256, loss = 0.8950)\n",
            "Saving model (epoch =  329, loss = 0.8945)\n",
            "Saving model (epoch =  404, loss = 0.8939)\n",
            "Saving model (epoch =  420, loss = 0.8934)\n",
            "Saving model (epoch =  564, loss = 0.8929)\n",
            "Finished training after 765 epochs\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "hsNO9nnXQBvP",
        "outputId": "9b1abe4c-1048-4b16-c9ee-58d0a249cac0"
      },
      "source": [
        "plot_learning_curve(model_loss_record, title='deep model')"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUZf7A8c83vQKB0ENviSAgxS4KZ0HsvZy9cOopeHd2z7Od5dTf6amnqKdynmKXswGWQywgIig91FASCCUJaaRnn98fM7vZJLubTdlssvm+X699ZXbmmZnn2d3Md57nmXlGjDEopZTquMKCnQGllFLBpYFAKaU6OA0ESinVwWkgUEqpDk4DgVJKdXAaCJRSqoPTQKBahIgcJyIbg52PtkJEjhGRzSJSLCJn+5F+toj8tTXy1lpEZJGIXOdnWiMiQwOdJ+WZBoIQICLbReTEYObBGPO9MWZEMPPQxjwEPG+MSTDG/DfYmVHKFw0Eyi8iEh7sPDRXK5dhALCuFfenVJNpIAhhIhImIneJyFYRyRWR90Skq9vy90Vkj4gUiMh3IjLSbdlsEXlRROaJyEFgsl3zuE1EVtvrvCsiMXb6E0Qky219r2nt5XeISLaI7BaR63w1DYhIVxF53U57QET+a8+/SkR+qJPWtR0PZbjNLm+4W/pzRGS1P5+Xh3xdLyJbRCRPRD4RkT72/K3AYOBTu2ko2sO6h4nILyJSJCLvAjF1lp8uIitFJF9ElojIaLdlfUTkQxHZLyLbRGSG27IHROQD+/MusvcxxkcZjIjcZDdjFYnIwyIyxN5nof0ZRDVUZnvZSSKywf6+nwekzr6uEZF0+zv8QkQGeMuXamXGGH218xewHTjRw/yZwFIgBYgGXgLedlt+DZBoL3sGWOm2bDZQAByDdcIQY+9nGdAH6AqkAzfY6U8AsurkyVvaqcAeYCQQB7wJGGCol/J9DrwLJAGRwPH2/KuAH+qkdW3HSxm2Aie5pX8fuMufz6vOfqYAOcA4O+1zwHcNfSf2sihgB/AHuzznA5XAX+3lhwH7gCOAcOBKe3vRdjlWAH+xtzMYyABOsdd9wN7W+fa2bwO2AZFe8mKAj4FO9vdRDvzP3m5nYD1wZUNlBpKBIrf9/gGoAq6zl58FbAHSgAjgz8AST9+bvoJwDAl2BvTVAl+i90CQDvzG7X1v+yAR4SFtF/ufsbP9fjbwhof9XOb2/glglj19AvUDgbe0rwGPuS0b6u1AYOfZASR5WHYVDQeCumX4K/CaPZ0IHAQGNOHzehV4wu19gp12oK/vxF42CdgNiNu8JdQEgheBh+ussxE4His47Kyz7G7gdXv6AWCp27IwIBs4zkteDHCM2/sVwJ1u7/8PeKahMgNX1NmvAFnUBIL5wLV18lXi9tlrIAjiS5uGQtsAYK7dvJCPdaCrBnqKSLiIPG43gxRiHbjAOrNzyvSwzT1u0yVYBwNvvKXtU2fbnvbj1A/IM8Yc8JHGl7rbngOcazfXnAv8YozZYS/z+nl52G4frLN6AIwxxUAu0NePPPUBdhn7CGjb4TY9APiTMx92XvrZ6w0A+tRZdk+dPLrKbIxxYB2Q++DdXrfpUg/v3b83b2Wu9Z3aZXP/7AcA/3DLcx5WsPDn81IBFhHsDKiAygSuMcYsrrtARC7Hqq6fiBUEOgMHqN2uG6ihabOxml+c+vlImwl0FZEuxpj8OssOYjUtASAivTysX6sMxpj1IrIDOBW4FCswuO/L4+flwW6sg5tz3/FAN2CXH+tmA31FRNyCQX+sZitnPh4xxjxSd0UROQrYZowZ5mP7/dzSh2F91rv9yFdDfJU5u85+hdrfq7NMb7VAPlQL0xpB6IgUkRi3VwQwC3jE2SknIt1F5Cw7fSJWe3Au1sH00VbM63vA1SKSJiJxwH3eEhpjsrGaFV4QkSQRiRSRSfbiVcBIERlrd0Q/4Of+52D1B0zC6iNw8vV51fW2XYaxdu3iUeAnY8x2P/b/I1b7+Qy7POcCh7stfwW4QUSOEEu8iJwmIolY/S5FInKniMTaNbtRIjLRbf3xInKu/Ru4Fet7XupHvhriq8yfY30Xzv3OANwD8yzgbrEvSBCRziJyQQvkSbUADQShYx5WNd75egD4B/AJ8KWIFGEdDI6w07+BVc3fhdUh2BIHCr8YY+YDzwLfYHUgOvdd7mWVy7HaojdgdaLeam9nE9b1+l8Dm4EfvKxf19tY7e0LjTE5bvN9fV51y/A1VgD7EOtseAhwsT87N8ZUYDVLXYXVRHIR8JHb8uXA9cDzWLW0LXZajDHVwOnAWKxO4BzgX1g1OqeP7W0ewPrszjXGVPqTtwby7bXM9ud4AfA41snFMGCx27pzgb8B79hNkWuxamWqDZDazZRKtT4RScM6MEQbY6qCnZ/2TEQewOp0vSzYeVHth9YIVFCIdf1+tIgkYZ0pfqpBQKngCGggEOumojX2jTHLA7kv1e78DquZZyvWlTk3Bjc7SnVcAW0aEpHtwIQ67bBKKaXaEG0aUkqpDi7QNYJtWFcuGOAlY8zLHtJMB6YDxMfHj09NTW3y/vLyC8gMj2TQ7iw6jRje5O0opVR7sWLFihxjTPfmbCPQgaCvMWaXiPQAvgJuMcZ85y39hAkTzPLlTe9K+Cgji5t25DD7wT8xddH/mrwdpZRqL0RkhTFmQnO2EdCmIWPMLvvvPmAutW+aaXHhCfEAOERbvJRSyl8BO2Lad0MmOqeBk7GuFQ+YMOfoCOI7nVJKqRqBHGuoJ9YAXs79zDHGLAjg/gizA4DWCJRSyn8BCwTGmAzA6wMxAsFZETCiVQKlOorKykqysrIoKysLdlYCKiYmhpSUFCIjI1t82yE1+qizacho25BSHUZWVhaJiYkMHDgQCdGTQGMMubm5ZGVlMWjQoBbffki1oTh/A46w0PwxKKXqKysro1u3biEbBABEhG7dugWs1hNagcDDlFIq9IVyEHAKZBlDKhCE2R+UowP8KJRSqqWEVCBwdRZr05BSqpXk5+fzwgsvNHq9adOmkZ9f96F7wRFSgcBZGO0sVkq1Fm+BoKrK96jq8+bNo0uXLoHKVqOE1FVDrs5ibRpSSrWSu+66i61btzJ27FgiIyOJiYkhKSmJDRs2sGnTJs4++2wyMzMpKytj5syZTJ8+HYCBAweyfPlyiouLOfXUUzn22GNZsmQJffv25eOPPyY2NrbVyhBSgaDmzmINBEp1RHsefZTy9A0tus3otFR63XOP1+WPP/44a9euZeXKlSxatIjTTjuNtWvXui7zfO211+jatSulpaVMnDiR8847j27dutXaxubNm3n77bd55ZVXuPDCC/nwww+57LLWe8hcaAUCrREopYLs8MMPr3Wt/7PPPsvcuXMByMzMZPPmzfUCwaBBgxg7diwA48ePZ/v27a2WXwixQOCkdxYr1TH5OnNvLfHx8a7pRYsW8fXXX/Pjjz8SFxfHCSec4PFegOjoaNd0eHg4paWlrZJXpxDrLLYCQHifPkHOiVKqo0hMTKSoqMjjsoKCApKSkoiLi2PDhg0sXbq0lXPnn5CqETgrAuE9ewU3I0qpDqNbt24cc8wxjBo1itjYWHr27OlaNnXqVGbNmkVaWhojRozgyCOPDGJOvQupQOCs3pSuXhXUfCilOpY5c+Z4nB8dHc38+fM9LnP2AyQnJ7N2bc0I/bfddluL568hodU0pHcWK6VUo4VUIKgZhjqkiqWUUgEVUkdM153FWiFQSim/hVQgwHUfQWgVSymlAimkjphhOsaQUko1WmgFAteDaUKqWEopFVAhdcTUZxYrpdqCBx54gKeeeirY2fBbSAUC5+WjJsj5UEqp9iS0AoH9V5uGlFKt7ZFHHmH48OEce+yxbNy4EYCtW7cydepUxo8fz3HHHceGDRsoKChgwIABOBwOAA4ePEi/fv2orKwMWt5D6s5iJ20aUqpjum9zFmuLW3bAtlEJsTw8LMVnmhUrVvDOO++wcuVKqqqqGDduHOPHj2f69OnMmjWLYcOG8dNPP3HTTTexcOFCxo4dy7fffsvkyZP57LPPOOWUU4iMjGzRfDdGSAWCmqYhDQRKqdbz/fffc8455xAXFwfAmWeeSVlZGUuWLOGCCy5wpSsvLwfgoosu4t1332Xy5Mm888473HTTTUHJt1NIBQJ9ZrFSHVtDZ+6tyeFw0KVLF1auXFlv2Zlnnsk999xDXl4eK1asYMqUKUHIYY2Qakx3Hv+1RqCUak2TJk3iv//9L6WlpRQVFfHpp58SFxfHoEGDeP/99wEwxrBqlTUgZkJCAhMnTmTmzJmcfvrphIeHBzP7IRYI0EHnlFKtb9y4cVx00UWMGTOGU089lYkTJwLw1ltv8eqrrzJmzBhGjhzJxx9/7Frnoosu4s033+Siiy4KVrZdQrNpSAOBUqqV3Xvvvdx777315i9YsMBj+vPPPx9j2sbF7qFVI3A2DWkgUEopv4VUIHDSQKCUUv4LqUDgunxUA4FSHUpbaWIJpECWMbQCgf1XO4uV6jhiYmLIzc0N6WBgjCE3N5eYmJiAbD9EO4tDKr4ppXxISUkhKyuL/fv3BzsrARUTE0NKSmDukwipQFDTNBTkjCilWk1kZCSDBg0KdjbatZA6ddYagVJKNV7Aj5giEi4iv4rIZ4HeV83lo4Hek1JKhY7WOHWeCaS3wn4QvbNYKaUaLaCBQERSgNOAfwVyP6792X+1aUgppfwX6CPmM8AdgMNbAhGZLiLLRWR5c3v9tWlIKaUaL2CBQEROB/YZY1b4SmeMedkYM8EYM6F79+7N26dzm1ojUEopvwXyiHkMcKaIbAfeAaaIyJsB3J8+s1gppZogYIHAGHO3MSbFGDMQuBhYaIy5LFD7g5rCaI1AKaX8F1JHTGfTkEOfUKaUUn5rlTuLjTGLgEWB3o/oM4uVUqrRQqpGUNM0pIFAKaX8FVKBwHn8l65Jwc2IUkq1IyEVCJzPLNarhpRSyn8hFQj0PgKllGq8kDpiuu4sDm42lFKqXQmpQOC6fFQ7i5VSym+hFQj0mcVKKdVoIRUIAMKMA0dFRbCzoZRS7UbIBQIchury8mDnQiml2o2QCwRhxmBEMNXVwc6KUkq1CyEXCMQ4tI9AKaUaIQQDATgkDIxeRKqUUv4IuUAQZhzomHNKKeW/kAsEOGsESiml/BJyR8ww47DuLNamIaWU8kvIBQIxBhMWcsVSSqmACbkjpmB0iAmllGqEkAsEYQ6D9hYrpZT/Qi4QgP3MYu0jUEopv4RcILA6i7VGoJRS/gq5QCAOo3cWK6VUI4ReILA7i7VhSCml/BNygSDMmJqn2CullGpQyAUCcdiXj2pnsVJK+SX0AgHaR6CUUo0ReoHAaCBQSqnGCMlAoE1DSinlv5ALBNpZrJRSjRNygUCMQ2sESinVCCEYCMDo8wiUUspvIXfEtJ5ZHOxcKKVU+xGCgUCfWayUUo0RgoFAA4BSSjVGyAWCMOPQYaiVUqoRAhYIRCRGRJaJyCoRWSciDwZqX+7CExO1s1gppRohIoDbLgemGGOKRSQS+EFE5htjlgZwn4RHR+vIo0op1QgBCwTGGAMU228j7VfAj9ECmLAwbRlSSik/BbQNRUTCRWQlsA/4yhjzk4c000VkuYgs379/f7P36cjNxQDVuTnN3pZSSnUEAQ0ExphqY8xYIAU4XERGeUjzsjFmgjFmQvfu3Zu9T6mowCFhVGZnN3tbSinVETQqEIhImIh0auxOjDH5wDfA1Mau21iCwYQJOByB3pVSSoWEBgOBiMwRkU4iEg+sBdaLyO1+rNddRLrY07HAScCG5ma4wf0ag0EwDu0kUEopf/hTIzjEGFMInA3MBwYBl/uxXm/gGxFZDfyM1UfwWZNz6ifX8wiM1giUUsof/lw1FGlf/nk28LwxplJEGjzdNsasBg5rbgYbyxUItGlIKaX84k+N4CVgOxAPfCciA4DCQGaqOcLsB9MYDQRKKeWXBmsExphngWfdZu0QkcmBy1LzaI1AKaUax5/O4pl2Z7GIyKsi8gswpRXy1iQ1fQTaWayUUv7wp2noGruz+GQgCauj+PGA5qoZwuxAoE1DSinlH38CgfMxL9OA/xhj1rnNa3ucD6/Xy0eVUsov/gSCFSLyJVYg+EJEEoE2e7od5uojqA52VpRSql3w5/LRa4GxQIYxpkREugFXBzZbTSeupiGtESillD/8uWrIISIpwKUiAvCtMebTgOesiZyBoHzTJuC0YGdHKaXaPH+uGnocmAmst18zROTRQGesqcKMAyNCybJlwc6KUkq1C/40DU0DxhpjjdkgIv8GfgXuCWTGmsx+eL3RISaUUsov/o4+2sVtunMgMtJSrBoBetWQUkr5yZ8awWPAryLyDdZlo5OAuwKaq2YQwEgYYbGxwc6KUkq1C/50Fr8tIouAifasO40xewKaq2YQhwMDJExps6NgKKVUm+I1EIjIuDqzsuy/fUSkjzHml8Blq+kiEhMwEkZkjx7BzopSSrULvmoE/+djmaGNjjcUPXgwjux9RPYfEOysKKVUu+A1EBhj2mXbShiCacMjYCilVFsT0IfXB4PVWayBQCml/BVygSAMrMtHlVJK+SXkAoHz8lGllFL+8XrEFJHL3KaPqbPs5kBmqjnEUW09qrKiPNhZUUqpdsHXqfMf3aafq7PsmgDkpUVUbtsOIuS++lqws6KUUu2Cr0AgXqY9vW87qqusGkG51giUUsofvgKB8TLt6X2bEYbRq4aUUqoRfN1Qlioiq7HO/ofY09jvBwc8Z03keni9Ukopv/gKBGmtlosWJAbrmcVKKaX84uvO4h3u7+1HVE4CdhpjVgQ6Y00VhgENBEop5Tdfl49+JiKj7OnewFqsq4X+IyK3tlL+Gk2M0RqBUko1gq/O4kHGmLX29NXAV8aYM4AjaMOXj1p9BHpDmVJK+cvXEbPSbfo3wDwAY0wR0GafAxlmdIgJpZRqDF+dxZkicgvWcwjGAQsARCQWiGyFvDWRwSFhVO1ts8/OUUqpNsVXjeBaYCRwFXCRMSbfnn8k8HqA89VkYfaziss3bwlyTpRSqn3wddXQPuAGD/O/Ab4JZKaaIwyDI0zbhpRSyl++HlX5ia8VjTFntnx2WoB2FiulVKP46iM4CsgE3gZ+oi2PL+QmzDja7vgXSinVBvkKBL2Ak4BLgEuBz4G3jTHrWiNjTWXdR6A1AqWU8pfXI6YxptoYs8AYcyVWB/EWYJG/zyIQkX4i8o2IrBeRdSIys4Xy7Hu/xrSTuotSSrUNvmoEiEg0cBpWrWAg8Cww189tVwF/Msb8IiKJwAoR+coYs74Z+W2Q1giUUqpxfHUWvwGMwrqR7EG3u4z9YozJBrLt6SIRSQf6AgENBHGHjtbRR5VSqhF8nTpfBgwDZgJLRKTQfhWJSGFjdiIiA4HDsDqd6y6bLiLLRWT5/v37G7NZjyI7JWogUEqpRvB1H0GLtK+ISALwIXCrMaZeADHGvAy8DDBhwoQWueBHB51TSin/BbQxXUQisYLAW8aYjwK5L6cwQYehVkqpRghYIBARAV4F0o0xfw/UfuoKAxxh2lmslFL+CuQR8xjgcmCKiKy0X9MCuD9LtTUwqt5UppRS/vF5+WhzGGN+IAhX9DvyciGhu3YYK6WUn0KuDUWMVRfQQKCUUv4JvUBgD0Nt9PZipZTyS8gFgjBj9RHoUNRKKeWfkAsEOLRpSCmlGiPkAkFk1yRAm4aUUspfoRcIkpMBMNo0pJRSfgm5QOAskI5AqpRS/gm5o6WzQHpDmVJK+SfkAoGT0WEmlFLKLyF3tAzDeR+BUkopf4RgILAY7SNQSim/hNzR0nmtkN5QppRS/gm5QFDTWayBQCml/BFygcBJ7yxWSin/hFwgcLYIGRFKfv01uJlRSql2IPQCgX25kBFhxyWXBjczSinVDoRcIHB1FntpGjKVlZRnbGu9DCmlVBsXcoHAdR+Bl0Cw55FHyJg2jcp9+1ozW0op1WaFXCCIsANAVbjnp3CWLPsZAEdRUavlSSml2rKQCwRR9t/KSCsQ7Pzd70hPTaPk559rJzR677FSSkEIBoJIu0Wo0q4RHPz2OwB2XH5Fo7dVtn49O6dPx1RUtFj+AAq/+orqwkK/05euWUN6ahrl27RvQ3lXXVhIdfHBYGdDtUMhFwhcNYKIyGZva/e9f+bgd99TvmVLs7flVLlrF7tumUHWzbfgKCnxa52CTz4F4OD337dYPtqLqgMHqC4oCMi2jTEU/7AYEyK1w02HH8GmI47wO72jvFwvnFBAKAYCu4+gMqJ+H0Hua69TkZFhvanzz39w6VIcpaUet9mSBwrnPkqWLWPjuPEttl2niqwsir7+usW325Kq8vK8ftZ1bT7qaDYdcWRA8lEw979kXncdBR99FJDtB0V1td9Js++5l4xp06hug/1l1YWFpKemkffWW83ajjGGrVNPJf+//22hnIWmkAsEMXGxAFR4qBHse+IJj+scXLKEnVddTfb997vmmaoqytPTWz6DjQgqldnZFP+w2Oe6ZRs3cXDpUtf7jDPPIuvmWzxuL/PGm9h6+ukN7jf7/gcoXPCF3/n0R/4HH7j6aTYffQzbL76k2dt0lJQ0q7msctcu++9ur2nKt2whPTWNooULm7wff1Xl5mIcjhbbXt6cORxctszr8oPLfgLwOyg3RnnGNnbfe2+TT6Iqs/cAkP/OO7Xmm+rqBrdZuXcfBZ9+6npfsX072Xfd3aR8dBQhFwjiunUFoDLSd9NQVU4O1cXFlGdksPOaawGo2LLVtbx806Z661Tu20d6ahoFn3/udbvuP1JjDMXff197nqP2j9hRXo6jtBRTUUHhV18BsOv2O9gwegxbJk8h87rrqNi6FW+2nXUWO6+6msIFX5D7+myMj+am4m++qVVGb/LffZddt97aYDqrPJ4PXMU/LGbLlN+4DtTZf76vVj9N+caNrumiRYvImmntr+Djjzm49CeP28x99TXy59ac2WXdMoOMU6c1v8bmYziS0lWrrTx+5V8t68Dbb1P0zTeN2n11cTEVmZlsPuZYcl58kaqcHJ+1uqKvvyb9kJE4Dh6kuqgIx0HP/QJ7H3qYnVdc6X3HAWwRy5g2jYIPP6Js7dombsGZuZrvpjo/nw0jR5E3+98+18y87lp2335Hm6zptFUhFwiiXJePhvtMt/Pqa9g0YSLlGza45pWtX8/ue+8lPTWtdmL7N1lh9xXkvvQy+599jpyXXyE9NQ1HWRkABZ9/zoa0Q6jYuROAwk8+IfP66eS/977btmofODeOGcvGw8axYfQYdt0yg+LFiyn89NNaHdTuP+jKffvIe+M/gBVEnHbdeiv7/va3mt0YQ+GXX2Kqqnx+DpuOPY6cWS95XFb3LPjgT8soW7/e9b503To2HDKS4jp9F0Vff03mdddRuXs3GadO87l/gKwbbqToiy/YfMJkdt95Fzuvuspjun1PPkn23TVndgcXL/aYzhNTUVG/09+vAFI/TVVODlkzb2XH5VeQ/Zf7KVm+nLw3rSaMPQ8+RNaNN/ncYuGCBRR++SWVe617WTZNmMjWqacCUPztd+y8fjpZN9/Cvn/8A+OhqWf/s8+Bw0HZxo1smng4m4+b5Ec5vBMPgbAqJ8d1wlNdWEjpmoYP6CXLl7PpiCNr/V5NZWXTMuX8bkTIeell0lPTqMjMBKBg7lyfqzo/V1qwdtVYjooK13GhPQi5QBBtP5nMU9OQJ7v++Kda7ws+tNuLPfxzVBcVA1ZtIeeFF9j/978D1j+/cTjY++hjAGw9+RTSU9OozM62lrs1OTV08PF0Jli2Zk1Nfm+Zwd5HH6Vixw4cxcVet5Nx6jR2zZjJ9osupnD+fLLvu8+1zFFRQZl9Rl6dk8P+Z56ptcwp66bfu6YrsrLYeeWVbDv3PPI/sv4RS1f8AkDxom9r7btu05R7Z3v2fX9xTZeuW0e5s88GqNqzx2t5fHL7TIv+9z/KMzIoWriw1pn5hnHj2TB6TJ3LiGsONk6FC76gcP58AEp+/ZV8+/dQMHeuqwll//PPU/TFF5T8/DP5773HjssuZ+9f/+oxa8Xff09FVlatebtu/QO7Zsxky/HH18x0HvAFKu0TidwXZ7Fp4uGuJLmzZ1M4b56rvDsu/S1gNZG5Nw+2hMzpv2P3n26jOj+fnddex/YLLmhwnf3P/5PqgoJav1f376Zs4ybyP/ig3npVubkULVrEwR9/rKndONcLCyP3X/8CwHGwxDWvqSqzsylZvrzJ6/try5TfsHHsYQHfT0vxfNdVOxYZ5uwsbt5VQ+5nMgcXLyZ21Eh2zZzpMW3Z+vXs+7//ozo3t/b8detc046SEhBh2znn+txv/tvveF1W8utKqvPzAdjz6KP0eeQRr2krtm935WHXH/5Ya9nG0WMA6HLB+a55B39aRu5rrxIzfHj9bWXtYuuJJ7neZ99zD9EjhtccVEXY++STdDrlFGJHj663fsbpZ9SU7/2a2tH2886vl9ad+xnV7nvv9Zou6+Zb6HnP3USlpJD1+5trLRu2ZDF7HngQ7JrRnof/Sq8HHyB2zBhyXngRgNJffiE9NY2BH37gahKr+5kB5Lz8Msk33ui1ScW9Jlmdn094ly5kXj8dgKH/+5rIvn19ltcT9yvL9j1u1fiihw2tl27nVVf7t72yMoq++opOp5/u86Sk0g7Kprq69oEdOLhsGTuvuJJeDz9El3PPRZy1by8tbI6yMhwlJWw76ywAupxvfe9VublUFxSya+ZMyjdvBiDxlFPoPuMWqvbvt7cpNfm0a9PiZyBwlJUR1qlTrXlbT52GKSsjbUPt/r/cV1+lYmcmvR98gPKMbWRMm8bAd98hdswYv/blrrqggOqcnEavF0whFwiinVcNRTavaNsvvMg1vf/pp9n/9NNe05Zv3FirzdvJvV25Ja4QKlqwwDXtvD+iOfLfrzk723nllR63W5mdzdYTT6y3bvY999Yqc96rr5H36muEJSQ0O18A+R/NZf/zz7neu2pqzveffuaaLl64kOKFCxn245J629l89DG13pdv2sSOSy5l8Lyafp6DS5bU+utNyY9L2fjiLL/yv+evj9Rqu8/6wx8Z9BCW6wsAACAASURBVN67fq1b9wC997HHah3Qyjf7vpy58Kuv2HXLjFrzyjZtwlFUROHn8zgwZw67b7/Ddx6ctSS3vBR8/jmdTzvN1e+w576/sMeu4Y1YWTPSr3ufzZ6/PuL1oostJ56EqdNRXZGxlYzT3C5okJo87Lz6GoB6w8NU5eRQuXs3pWvW0OW881xBf8vxJ5C6rqZJy1FWhqnTXFORtYvInj3Y9+RTAHS9/DLX76Dgs88bDASmupqy9A0UzptH8k03Url7NxU7dtTss6KCzUceRfeZM+h6pY/+miALuUAQ1UI1gvZg87HHBXwfWyZP8TjfPQgcePNN17Sv5qrGyL7nHq/L8t58y2NTzOajjvZ7+xnTTqs3rzon10PKGqUrV/q9/cLPPqv1vmz1anJfn02nqaf4XK/M7px2l/fvN/zeL1AvCJSuWcP2Cy70voKPznLnARJg959uczUH1lVdUIApsQ/qbnHMUxAoz8gg7/XX6wUB8BDkqqrrdYZX5+RgHA5XzWDbOee6ahDlmzfXqkWVrFhRK4+1tlNYyNYTT6Tz+ee55mWcfgYJU+r/5vM/+IDsP99H32eeIWbUKCJ7dOfgzz+Tee11rjR5r70GQPzRNb9DZ+1772OP0/XKK60a2ZdfUpWTS2RKXzqdfHK9fQWDtKWbaSZMmGCWN7P9rqTaweDvVjN97hwu+fLThldQKogGf/oJGWecGdQ8JP/+98SOGU3CpEmYykr2P/sseW/8B+N2MUJDki67rNYJgS/Rw4Y2WKtpSOyYMQx812pGrXdxhxdDF33DlhMmW+uPG0fFtm1UHzjgNX3S5ZfTfcYtOEpL2TLJ6s+JHTu2UScE7tI2pLPtggtrNbWNWLWSsOjoJm3PSURWGGMmNGcbIVcjiHbWCLwMOqdUWxLsIACQ889/Nnsb/gYBaLhpyx+lq1aRnppG8s03N5zYye0qotJfPNds6qZ376yHxtUKPanb37L9kksY3AZuaAy5q4bCRQivrqKsmVFWKdX25Tz/vN9pt0z5TaO2faCZdzXXVeahH7F8fTpZt8yo1ecVDCEXCAD65Owjq0fvYGdDKaVctp11tsf5RV99xe7bb2/l3NQWsEAgIq+JyD4RaeqthU02aFcmW1MGtPZulVKqXQpkjWA2MDWA2/dq7Kb17O7ek79M/0Mwdq+UUu1KwAKBMeY7IC9Q2/flpGU/APD9YYdz//W3sr9zUqvtuzQ6mt3JPVptfyq4CuIT+fzoE4KdDaWaJeh9BCIyXUSWi8jy/c67CZspobSEv75oXf/83bgjuPv3d1BuD0JXLcKersnkJXbm82MmN2n7Btjl5WB/58138duH/xHI8bxUG/Lo1Tfx1OW/Y2fP9tsnlZ+QGOwsqCALeiAwxrxsjJlgjJnQvXv3Ftlm4qlTOWb1Cp5/wrrrcWu/gZz51L8oiY7hlXMu4ZJHnuOq+5/iqcums6drstft5HXq7PHsft7RJ3DZw/9g3aBh9ZatGZoKwMGY2BYpize+Ak1pdDSZPXpRLUJpVOOunnKIkNupS615e5O6sbH/4CbksvleOO8y1g6u/zm3tPyERPYmdeOxK2+s9Z0Xx8T6/C6zu1m/2ZIAf9+B8uOowzjnyZdZOcy/a/EDoTQqmlfOuphSD1f6VYtQ7OWzLYyLb7ETrnWDhlEcG+czzRdHHMeck8/kh9ENjxJQ2cCgl/kJiRTEt50AHPRAEAhdzrXG8xm5bTOvPnwHg7N2UBEVxfmPv8C7J1nj3hTFW0MhXPLIcxisA2uBPW9bnxQOxsRy3t9m8duH/8Gu7j3J7dSFZYeM4cvDj+Wpy38HwM13PMTLZ1/MrHMu5Z2TTnfVOgBeO/MiMvr0446b76IkOoalI8eyK7mH66BSLeJK//bJZzD3+JPJT0hkeeqhFMbFA/D8BVfw8SRreIeC+AT+fdp5lEVGMe+oEzjl2X+zyg46dT18zS1c8eDT3PrHvzDtH7OZe/zJrBs0jNVDR5CX2JmdPfuwpW9/j+t+etxvOP9vL7I5ZSAAG/sP4uJHn+eGux9x/dPt6NWHVUNTqQoLpzgm1vUP+cHkqVx/z2Pk2E1xBfGJ9YKKu4qISJ6++Bqu/MtT7OmazJa+/dnWJ8W1vCA+gfdPPI1bbn+INUNGANaB682p1pg1nv5xq0VqHcjLIyMpiY7xuP9qtztqz/3bLC5+9Hm+PHIS9954Gz+OsgYMO/eJWVz8yLNey+Cw727NT7CGgCiNim7wIGCVLZElh47jy8OPxWHn4+mLr/HZr1UQn4BDhC8PP5YtXi6GWDj+SPYmdQOs33RpdDQvnvtbiuLieeq31/POSbWfR7E87VAA0gfWH78IrM/viyOOwyFC+oAhPr/P104/nwsefb7ewbk0KppnL7yKNYOHc8X9T7Gp30AAlqWNZu7xJ/PJpBOZM/UsXjnrYu698TbyOnV2rfvKOZdwxtOv1frfAtiX1JWz/u9ffDTZ953aACXRMa7/qbzEzrU+u7zEzuzpmszNdzzEGX9/1euJU3VYGI9fdROvnHMJ9914m+s7K4yL569X/57HrryRXck92N85iVnnXMrJz7/JgiO9jwp73uMvcvZTLwPw6pkXBjUQQ4DvLBaRgcBnxphR/qRviTuLAar27681NG95ZCT/Pf5kVg1L40cP0TymvIyI6iqK41pmnJyGjNy6idjyUpYfMoYj1v7KT6Pqj1I4453XefZiayCx6XPn8NbUszno5YwlddsWRmVsoiIikh8PHcf+rt38ykfP3P0cvXoFA/bsIqtHb45ZtZw/zbwXR3g4I7du4orPP+TOGTXDPh+z8mcun/9fbrjbGuyuR14O++wa1amLv2G+3dSWum0LU5Yv4bUzLqQsJoZnn7yfd086HSNCfGkJiSUHWZF2KI6wMDJ79qmXrzO++5oRO7by5RGTWD285h/kvn/9g4evqz3wX++cvRyzcjkfnHgaYzatp1tBPgsnHs3N781maOYOnrj8d+zu0Yvpc+ewcvgh9MrZz2Xz57J2yAgeueb3XPXZB0RWVTHrvMt8flZv3jeTcIeDXd178uPo8Rz/y0+URUVxxwxrKIzjVyzluJU/8+pZFxJRXc0Tzz7Gtj79SCwpptPBg/xv4tEsOOp4nn/yfjb2H8RbU89mwyDr4HvPa88TVVXJA3YQmLpkERWRkdz6zuukDxjCoOws/nbFDaywD9pON3z4Jj8fMgYxhnEb1vLzIaP5NXUUvXL2ceHXnzH79Aso9NDsc8u7s0ndvoWChE7c83trzKE++/Yw853X2dG7Lz3zclg2ciy5nbuw9NBxAFw+7yP+M806wTpv4XwOX7uS3rn7+Pj4k8nq0Yv7Xn2O05+2hli4a/YLjNq6kXdOPoOq8AgW1OlDST6QywkrlvLBidYwH+d//blrGuDMb78ibfsW5h99AqvtA2T3vFzO+2Y+3fPzyOmcRGF8Am+deg6diwqZc99MvjhyEjt79aEyIpLw6moye/Xh/lesUXVvuf1BMnv2oXNRIQWJVsA+/fuvyerRm5UjRtbKW5/9ezn+l6UcSOxMRt/+xFSUk7ZtC++efAZ1hVdX0f1AHnvsE49Dt2xwtQg4Oct27MqfGZq5nb1dk1mRdqjr/2bE9q1sHDiE6XPn8NCznh+c1ZCWuLM4YIFARN4GTgCSgb3A/caYV32t01KBoHLfPtct4XU5RPhp5FjStm/h64nHsGnAYHYn9yCvcxeyk3vW+rE05PgVS/l2fOMeozggO4sdvVMaThhkMeVllHk5k1aqrvjSklonKmEOh6u2VFdEVRVVHh4l2xGFVVcTW17G7Idu57iffA966E2bHmLCGNP8ZxEGQJgxHLXWGinx/G8W1FtusAY8LI+MJLqykj1dk+lamI8RIaqyko0DBtM7Zz+dDxZhgJ2fvU+PvFzWDE2lV+4+SqNj6XSwiF65+1k7eDhF8Ql0Li5i5LbNHIyJJa6slJ/TRrOrR08O3bKRg7FxDN6daZ0lp46iWsJIKD3I3q7dGZ65jX1J3ehWcIDq8HBiy8qIrKpiX1I3Bu7J4pcRo0gqKiC8upo+OfvY1G8g3fMPsG7wMKYsX8L23v2ILS+lIiKS5IIDrBqWxvCd20gfOIShWTtJLClm9dBURuzIoCIyig0DBhNXXsZhG9chxvBL6igOJHbi8HWrKEhIpCoigj1dkylI6ER1WBj99mUzevMGKiIjWTT+SEbsyCCurIzvx05EjKEsOpp+e3aTUFrCL6mjyOvUmaGZOyiMT2BY1nb2JiUzecWPlMTEktepC2uHDGfEjgzCHQ5WDUvDIULv3P2M27CWJaPHE1NRTllUNGIc9DiQS1FcAr+OGMnE9avotzeb5WmHUhifwJFrf2XpqMPodLCYpKJCqsPCCHc4OGzjWvITOrF+8DC6FBXSf89uvh1/BOHV1Ry5diU7e/VhS8oADiRaTRMnLfuB1UNTSSoqoCo8nIqISIZlbmfDwCGEOxzkJXYmbcdWV/o++/eACAdjYkkfOJTDNq5j5YhDSNm3h/yETqTsyyazZx+29UlhwJ7dnPTTD3x32EQK4xM4ZNsW8hM6EVldRVlUNBWRkURVVlAcF0/3A7kUJHTisI3rqIyIJNk+K97epx+/jBjJYZvWkT5wKPFlJfTO2U9lRARDsnYQU17OL2mjmLB+DRl9+xPusGq9h2RsYuXwkfTfs4ttffoRX1ZKdrfuDM3aQXRFOQmlJWwYMIS48jJiyssYlrmdX0aM5Kg1v7J6WCr99mazL6kbVRERxJaXseDISfQ4kMthG9eTnJ/HktHjqQ4PY+TWzcSVlxJTXs7+pK5s7D+YCxbOI2VvNjt6p7B28HAcYWEc6NSZfUndSN2xlQHZu1g8ZgL7u3RFjKFPzl7iysqoiohg+I4MiuPiCXM4yE7uTnL+AYri4slP7ETq9q2MzNhMaXQ0Cycew4HETlRERtE9P49x6WuIqK4mOT+PTQMGM3xHBpv7DyI5P49hmdspj4wiprKCbX1S2JfUjeqwcOJLSxi+cxs7e/VlZ8/ehBlDz7z99MzL5UBiJ7KTe5BQWsLu5B70yt3P8J3bSR84hH1J3Th8/Sq2pAygMiKSfvuyKYqLR4yhIiKSvM5dGLM5nQOdOjNiR4arebPzwZYZrLGpQm7QOYCqAwcaNRKlUkoFW91nJPirJWoEIdlZHJGURJ8nnwx2NpRSql0IyUAA0PmM0xtOpJRSKnQDgVJKKf+EdCDodFr9p1AppZSqLaQDQZ/HHmXQRx8GOxtKKdWmhXQgkKgoYg45JNjZUEqpNi2kA4E3nU47jZhDD/W6vNv119H/jX/Xm9/lkoubve+Y0aObvY3oNP9vR+9ywQW13g98/32/1gtLbN44KD1uv52h3yx0vR+2ZLHXy+NGrPy1WfvyV/KMW1plP0q1Nx0yEPR+7FEGvf8eQ75YQNcrr2DEiuW1DtBhiZ2IP7z2s0oHz5tH7/vvp//s2TUz3caq8Ud492QGvfduvflRgweT8vxz9Hv5JddBcfBnn7qWux9AI3r1IslHQOr914dJ25DO0O++ZdjiH5CYmruDU/75PLGH1h/to4f9dCT3g7+ndK78DvD+0J+IPr1J25BOt2uvIbJ3zYicEV271ksbmZJC91tvJSwmhtT09STfdKPX7QIkTq15vEXquoafd5S2IZ1+L81yvY8daQ0n0P0P3sfz6fnnP7umez/+WIP78GXEqoafb5t06aXN2kcgpLzwAmEJjR9uJdzDdwzWiUtzTyxUYHWIQNDvpVkMnjfP9T4sKgqwDmg9776bsPh4Br33LlGDBlkJ6hzfk2++mejB1rL4I49wzU9LX18zvSGd1LVr6H7rrV7zkXTBhfXmdTrtNIbM+5zEE08kYdIkwmJiSNuQTvRQz4OADVv0DfFH19wsF3/00fS4/TYGzJlD9LBhrg7yyB49iOjWDbFv8+9x550k/sZ6ZmvfZ56h65VXurbR7dprGPHrLwz74XtiRo8mYcoU+j79NCkvvEDahnS6//GPNf/kEREM+WIBve63RnbtctFFJJ50Yk1+jjqqVn4Hvvcu/V552WNZ4o86iuQbrAH8RISw+Piaz+ryy+ulT5g0idTVqxi2ZDFSZ2C3Hnfe6XEfCccfT+qa1QxfvpyE449nyIL5JP9uumv5gLfeZPD8eQxb/AO9H3uMpEsvYcSvvzBi1Uq6nO350YL+Cmvgudnxx0+i5113MuA/bzD4s0/p87fHGTzv83rpwrtYA73FTZxIRO/GD3cdO94aX6vv038HrGBdbx9duzL4s0/pduMNJEw+geHLfvK8MS8nP/1nz2bYt4s8Lhs89yNG/LyM/v/+N93s7xtg2A/fu6YjeriN8ltnH+Hdkxn2/Xee8+Om9yOPkPL8cw2ma6w+TzZtDKDGSPmn/89eDoQOEQgSjj+e6MGD6Pvcs/R9zvtIkn2eeMKVHiB2zBiSb7qJ7jf/vla6/m/829UJPWLFcoZ++y0AEhHhOrBZM2p+0FFDh9DlwtrNNABdr7naa356PfQgvR56sN78qJQUEk44AYCky35Lt2uvJW7cYQz+9BPCYusM2esc78XhcM3qNPUUet59V+1ksbGERUcz6L136ffCPwnv3JnEKdYgcsnTr2f4ksUMX/ojw3+0xkPpdNppxB15JMm/m06fp55iyNdfMeTLL+h9//21ths7ejQJxx3net/vX/9yTXc5/7zan8Xll9Pj9ttIXbOanrffVmvZsCWL6XLuOUhUlKt20e3GG1zLu119Va30fZ952jUtkZGEJ1hBJmrgwFrp4saPJ3rQICK6daPLOWcjYWGuzwLw2EQIVuCP7Nu3Zv/XX0fns850vXedAdujZqauXsWQBfOJHjGi1nYkKoq4iROJHjqUzmedRfTgwUhcXK28xk2cSPLNN9P3H8/Q6957XOt2ueTies1tA9560zWv07RTSduQTtTAAXaeOpG2IZ1hCxcydNE3rubRqIEDGfq/r4keOpQeM2ciIq4TiHpEau0zMiWFtA3pxB95BFJnhFCgVnnjjzic5BtucJU7IjmZ/rNn0/Mv99UKgGnp6+n1oPW77/PkEwz64EMiuncn1e3EC2D40h9rnRR1Oe9cEk88kcRT6z8Ysdt119ZqUnXfVkSPHiT8pubB9u7lG/jO27Vqto0R2a9frW3W/e25C4vzPQR2oHWokZ86nXSSz+Wxh46q/SN49x2P6dybjcLi42udybrr87fH6XzmmfXmS1wcUf360eO2P7maKzxJurCmBpEweTLRw4e7bcTPZilnOuPwnc4PzjNTgPBOnRgw+3XX+6gU/wbSSzj2GAZ98jEiQvSw2s8ZkKgoul17ret92oZ0tvzmRCp37SLcQ9NCj5kzqT5wgPx3aje39XtpliuY++JPX0v84YdbTTzGsOf+Byj4+GMi7bL2n/06RV9+RZfzzyO8szXeUMHHn1jrHWkPRugMwCJEDRxIwpTJlG/cCEDsoZ77i4Z+/RWO4mIcpaVsO+ts4o46kq52E1LiiSfSf/brlK5ZQ/L111vlfeVlMq+3ajlx9tn/iJW/ug7MPe++h5hDDiH+mJqDZmSvXvR7aRZZM2bQ9+9/r38CUUfiqVMpmr/AdaNmZP/+VO7cSUxa7dE2e95zD3sffRSwDm6DP/5vreViDzaXdMkl9ud0hKuWPWLVSsT+vSZddCFJF9WuQUvdmkKXLvR7+SV23XY7nc+sGR20zyOPsHH+AiJ69qRq716ihg6hx223QcQzlKen033mDESEbjf8jtxZL5Hy4gvEjhxJemrN76H7rTOJ6t+f2LFjvX4mkf36UZmZ6Xoff9xxHPy+ppbT447b2XXLDNf7IQvmA9TaT5thjGkzr/Hjx5tQUJaRYQoXLgzoPir27DW7/3K/cVRU+ExXvnOn2Tx5iqnIzq63bPsVV5r1I1IDlcUWUbF7t8n/5FO/0q4fkep3eaoOHDDVpaWNzk/uG/8x5ZlZXpdX5uaaokWLXNsu+uEHk3nLDONwOIwxxjgqK03x4sWmdONG46iqanB/FXv2utb1ZduFF7X4d1ldXm5K1qw1m06YbErT003VgQPGUVlpLSspMfmffGKqi4vrrdfQ9+CoqPCrTJ44t+1PWR0OhymYN8/1P1K6YaNZPyLVlG3a5HXb2y6+xPO2ystN4f8WuvZ98OefTWVenmub60ekmurycuOoqDDZjzxi1o9INcVLfzLrR6SaHddeV2tbmTNvdaVxvnz9phoCLDfNPPaG5KBzyj+OsjIcRUVEtNCT4YJt0zHHUp2b2+TBu9ozR0kJ1fn5RPap/3yH1paemkbUoEEMmT+v4cRN2LZTS3/PVQcOEBYX57Nvx7l/932X/PorEUlJrqYfR0UFxd8sotMpJ+MoLUUiI101IXfF33+PREUTO2qk11YFf7TpYahV2xcWE0NYTOg8c2DIF19gKsqDnY2gCIuLC3o7s9OAOXOIGjQwoPvoed+fG07USBFJSU1aL+6w2g+WCouKotMpJ1vTPprc3PvOgk0DgQoZVodw08+sVMuIG1f/iXstretvfxvwfXgy/Kel0IZaUVqKBgKlVLuR8uILRPX3/Lzt1uC8KCDUaCBQSrUbiZMnBzsLIalD3EeglFLKOw0ESinVwWkgUEqpDk4DgVJKdXAaCJRSqoPTQKCUUh2cBgKllOrgNBAopVQHp4FAKaU6OA0ESinVwWkgUEqpDk4DgVJKdXAaCJRSqoPTQKCUUh2cBgKllOrgNBAopVQHp4FAKaU6uIAGAhGZKiIbRWSLiNwVyH0ppZRqmoAFAhEJB/4JnAocAlwiIocEan9KKaWaJpA1gsOBLcaYDGNMBfAOcFYA96eUUqoJAvnw+r5Aptv7LOCIuolEZDow3X5bLCIbm7i/ZCCnieu2RVqeti/UyhRq5YHQK5On8gxo7kYDGQj8Yox5GXi5udsRkeXGmAktkKU2QcvT9oVamUKtPBB6ZQpUeQLZNLQL6Of2PsWep5RSqg0JZCD4GRgmIoNEJAq4GPgkgPtTSinVBAFrGjLGVInIzcAXQDjwmjFmXaD2Rws0L7UxWp62L9TKFGrlgdArU0DKI8aYQGxXKaVUO6F3FiulVAengUAppTq4dh8I2tMwFiKyXUTWiMhKEVluz+sqIl+JyGb7b5I9X0TkWbtcq0VknNt2rrTTbxaRK1u5DK+JyD4RWes2r8XKICLj7c9oi72uBKE8D4jILvt7Wiki09yW3W3nbaOInOI23+Pv0L5Y4id7/rv2hROBLE8/EflGRNaLyDoRmWnPb8/fkbcytcvvSURiRGSZiKyyy/OgrzyISLT9fou9fGBTy+mVMabdvrA6obcCg4EoYBVwSLDz5SO/24HkOvOeAO6yp+8C/mZPTwPmAwIcCfxkz+8KZNh/k+zppFYswyRgHLA2EGUAltlpxV731CCU5wHgNg9pD7F/Y9HAIPu3F+7rdwi8B1xsT88CbgxweXoD4+zpRGCTne/2/B15K1O7/J7szy3Bno4EfrI/T495AG4CZtnTFwPvNrWc3l7tvUYQCsNYnAX8257+N3C22/w3jGUp0EVEegOnAF8ZY/KMMQeAr4CprZVZY8x3QF6d2S1SBntZJ2PMUmP90t9w21Zrlsebs4B3jDHlxphtwBas36DH36F9pjwF+MBe3/2zCQhjTLYx5hd7ughIx7rLvz1/R97K5E2b/p7sz7rYfhtpv4yPPLh/dx8Av7Hz3Khy+spTew8Enoax8PUDCTYDfCkiK8QaWgOgpzEm257eA/S0p72VrS2WuaXK0Neerjs/GG62m0peczaj0PjydAPyjTFVdea3CrsJ4TCsM86Q+I7qlAna6fckIuEishLYhxVkt/rIgyvf9vICO88tdoxo74GgvTnWGDMOa0TW34vIJPeF9hlWu76eNxTKALwIDAHGAtnA/wU3O40nIgnAh8CtxphC92Xt9TvyUKZ2+z0ZY6qNMWOxRlw4HEgNZn7aeyBoV8NYGGN22X/3AXOxfgB77eo29t99dnJvZWuLZW6pMuyyp+vOb1XGmL32P6oDeAXre4LGlycXq6klos78gBKRSKwD5lvGmI/s2e36O/JUpvb+PQEYY/KBb4CjfOTBlW97eWc7zy13jAhUh0hrvLDujM7A6ihxdoqMDHa+vOQ1Hkh0m16C1bb/JLU78Z6wp0+jdifeMnt+V2AbVgdekj3dtZXLMpDanastVgbqd0ROC0J5ertN/wGrHRZgJLU75zKwOua8/g6B96ndAXhTgMsiWO32z9SZ326/Ix9lapffE9Ad6GJPxwLfA6d7ywPwe2p3Fr/X1HJ6zVOg/8kC/cK66mETVhvbvcHOj498Dra/kFXAOmdesdr6/gdsBr52+2cTrAf7bAXWABPctnUNVsfQFuDqVi7H21jV8EqstsdrW7IMwARgrb3O89h3v7dyef5j53c11vhY7gece+28bcTtahlvv0P7e19ml/N9IDrA5TkWq9lnNbDSfk1r59+RtzK1y+8JGA38aud7LfAXX3kAYuz3W+zlg5taTm8vHWJCKaU6uPbeR6CUUqqZNBAopVQHp4FAKaU6OA0ESinVwWkgUEqpDk4DgWqzRKSb28iSe+qMNOlzdEgRmSAiz/qxjyUtl+N62+4iIjcFavtKtRS9fFS1CyLyAFBsjHnKbV6EqRmbpc2xx8X5zBgzKshZUconrRGodkVEZovILBH5CXhCRA4XkR9F5FcRWSIiI+x0J4jIZ/b0A/agZItEJENEZrhtr9gt/SIR+UBENojIW85x9kVkmj1vhVjj73/mIV8j7THmV9qDoA0DHgeG2POetNPdLiI/22mc49APdNtnup2HOHvZ42KNw79aRJ6qu1+lWkLAHl6vVAClAEcbY6pFpBNwnDGmSkROBB4FzvOwTiowGWs8+40i8qIxprJOmsOwbtvfDSwGjhHrAUIvAZOMMdtE5G0veboB+Icx5i272SocSdZ5zgAAAe5JREFUayiHUcYaXAwRORkYhjUmjgCf2AMP7gRGANcaYxaLyGvATSLyOnAOkGqMMSLSpfEflVIN0xqBao/eN8ZU29OdgffFesLY01gHck8+N9a47TlYA6719JBmmTEmy1iDmK3EGoMoFcgw1njvYA1J4cmPwD0icicwwBhT6iHNyfbrV+AXe9vD7GWZxpjF9vSbWMMqFABlwKsici5Q4mXfSjWLBgLVHh10m34Y+MZuhz8Da1wWT8rdpqvxXBv2J41Hxpg5wJlAKTBPRKZ4SCbAY8aYsfZrqDHmVecm6m/SVGHVHj7AGpRsgb/5UaoxNBCo9q4zNUPsXhWA7W8EBrs9J/YiT4lEZDBWzeFZ4GOsgcWKsJqinL4ArrHH1UdE+opID3tZfxE5yp6+FPjBTtfZGDMPa3TNMS1WKqXcaCBQ7d0TwGMi8isB6POym3huAhaIyAqsg3uBh6QXAmvtp06Nwnr8Yy6wWETWisiTxpgvgTnAjyKyButM3xkoNmI9rCgda9jnF+1ln4nIauAH4I8tXT6lQC8fVapBIpJgjCm2ryL6J7DZGPN0C25/IHqZqQoirREo1bDr7TP9dVhNUS8FOT9KtSitESilVAenNQKllOrgNBAopVQHp4FAKaU6OA0ESinVwWkgUEqpDu7/ATTyRzzoH50WAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "id": "3iZTVn5WQFpX",
        "outputId": "5d5b6d8b-cdb3-430c-ab39-9a4a34acae3c"
      },
      "source": [
        "del model\n",
        "model = NeuralNet(tr_set.dataset.dim).to(device)\n",
        "ckpt = torch.load(config['save_path'], map_location='cpu')  # Load your best model\n",
        "model.load_state_dict(ckpt)\n",
        "plot_pred(dv_set, model, device)  # Show prediction on the validation set"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAFNCAYAAACE8D3EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3iUZdaH7zPpISEJEDoBBQEVIih2V4qgoGLJ2v3sZdV17d3FxYJrXdfeCyquBVGxoKK0ta8iRASkE0ILJaT3eb4/zgyZhBAmkEk993XNNe0tz0T9ec5zmjjnMAzDMILD09gLMAzDaE6YaBqGYdQBE03DMIw6YKJpGIZRB0w0DcMw6oCJpmEYRh0w0TSCQkR6iYgTkfBGuPcqERnZ0PdtaKr/jUVkmohcsBvXSRGRfBEJq/9VGiaaTQgROUtEfhSRAhHJ8r2+SkSksddWG77/QP0Pr4gUBbw/t47Xek1E7gvVWvcUEblQRCp8vy1XROaJyImhuJdzboxzbmIQa6ryPxXnXIZzLs45VxGKdbV2TDSbCCJyI/A48DDQGegEXAEcCUTu5JwmYUn4/gONc87FARnA2IDPJvmPawwrNUR87/uticDLwLsiklT9oBb0e40ATDSbACKSANwDXOWcm+ycy3PKr865c51zJb7jXhORZ0XkMxEpAIaLyL4iMktEtonI7yJyUsB1Z4nIpQHvLxSRbwLeOxG5QkSW+s5/2m/VikiYiDwiIptFZAVwwm78rmEikikit4rIBuDV6msIWEcfEbkcOBe4xWfJfRxw2CARSReRHBF5R0Sia7hflO93DAj4LNln+XasdmwfEZntu95mEXmnrr/POecFXgFigN4iMl5EJovImyKSC1woIgki8rKIrBeRtSJyn/9/drv6G9fwz+8yEVkkInkislBEDhSRN4AU4GPf3+yWGtz8riIyVUS2isgyEbks4JrjReRdEXndd93fRWRIXf8WrQkTzabB4UAU8FEQx54DTADigR+Bj4EvgY7A34BJItKvDvc+ETgYSAXOAI7zfX6Z77vBwBDgtDpcM5DOQDugJ3B5bQc6514AJgEP+azUsQFfnwGMBvbyrfXCGs4vAaYAZ1c7b7ZzLqva4feif7ckoDvwZPA/SfGJ0qVAPrDU9/HJwGTUCp0EvAaUA33Qv+WxvnOgDn9jETkdGA+cD7QFTgK2OOfOo6p1/1ANp78NZAJdffe4X0RGBHx/ku+YRGAq8FSQf4JWiYlm06ADsNk5V+7/QES+81lNRSJydMCxHznnvvVZOYOAOOAB51ypc24G8AlVRWNXPOCc2+acywBm+q4JKjb/ds6tcc5tBf65m7/NC/zDOVfinCvazWsAPOGcW+dby8cB66zOW8BZAe/P8X1WnTJUyLs654qdc9/UcMzOOExEtgEb0L/1qc65HN933zvnPvT982kLHA9c55wr8An3YwHrq8vf+FL0fyb/83khy5xzq3e1UBHpgW7x3Or7nfOAl1Dx9fONc+4z3x7oG8ABQf4dWiUmmk2DLUCHwD0w59wRzrlE33eB/5zWBLzuCqzx/QfqZzXQrQ733hDwuhAV4e3Xrnbd3WGTc654N88NZGfrrM5MIFZEDhWRXqi4flDDcbcAAvzkc0kvrsNafnDOJTrnOjjnDnPOfRXwXeDfrCcQAaz3/Q9wG/A86hVA3f7GPYDldVijn67AVudcXrX7BP47Uv1vG237sTvH/jBNg++BEtS1e38Xxwa2pVoH9BART4BwpgBLfK8LgNiA4zvXYU3r0f9Q/aTU4dxAqrfRqrImEam+pj1qu+WcqxCRd1ELcCPwSTXB8B+3AXWPEZGjgK9EZI5zbtme3J+q61+D/nPtEOhFBFCXv/EaoHcQ96zOOqCdiMQH/B1SgLW1nGPUglmaTQDn3DbgbuAZETlNROJFxCMig4A2tZz6I2oZ3CIiESIyDBiL7k8BzAPSRCRWRPoAl9RhWe8C14hId19k+LY6/qydMR/YX0QG+YI546t9vxHYew/v8RZwJhpUqsk1R0ROF5HuvrfZqPB4azp2d3HOrUf3TR8Vkba+f6a9RWSo75C6/I1fAm4SkYNE6SMiPX3f7fRv5pxbA3wH/FNEokUkFf334M16+ImtEhPNJoJvA/8G1G3c6Hs8D9yK/ktf0zmlqEiOATYDzwDnO+cW+w55DCj1XWsiGpgIlheBL1CRm4sGWPYY59wSNFPgKzR4Un0v8WVgP587++Fu3uNH1KLtCkzzf+6LLv/J9/Zg4EcRyUeDH9c651b4jvtd6phfWgvnoyljC1Fxngx08X0X9N/YOfceGgB8C8gDPkQDbKB7oX/3/c1uquH0s4FeqNX5AbrH/FUNxxlBINaE2DAMI3jM0jQMw6gDIRNN3/7JTyIy3+fu3O37/DURWSlafjbPt29nGIbRLAhl9LwEGOGcyxeRCOAbEfHvL93snJscwnsbhmGEhJCJptPN0nzf2wjfwzZQDcNo1oR0T9NXWzsPyAKm+6KaABNE64gfE5GoUK7BMAyjPmmQ6LmIJKKpDn9DK1w2oGkYLwDLnXP31HDO5fhqldu0aXNQ//79Q75OwzBaKOvWQUUFhGljsFXbEtlSFEssvxQWOFdbLvQONFjKkYjcBRQ65x4J+GwYcJNzrtZ+hEOGDHE///xziFdoGEaL5eKLoXt3ylw45394Km8vGMg9Q7/mjdkjty1xboe2frURyuh5ss/CRERigFHAYhHp4vtMgFOABaFag2EYBgApKZRuzees90/j7QUDeXDkdMZ1e4VI7S5WJ0IZPe8CTPT1DvQA7zrnPhGRGSKSjDZLmIc22jUMwwgZJSf+mdNPLePjzP147NhpXNf/c5i1gGKoc+etUEbP09E+gdU/H1HD4YZhGCGhqAjSxg3k80x45vhPuLLTFEhKgb32ojQ9vaSu17MuR4ZhtFgKCuDkk2HGDHjpJbjkkhPRvs/A+PF4PvqoziNjrIzSMIwWSV4eHH88zJwJr70Gl1Tv8ZWWRjjUWTTN0jQMo8WRkwNjxsBPP8GkSXDWWTUclJrKeu0AVidMNA3DaFFkZ8Nxx8Gvv8I778Cf/7zzYwubUiDIMAyjodm8GUaNgoULYcoUGDt21+fUFRNNwzBaBFlZMHIkLFkCH/17BaN/eR0+yICUFEhLg9TUermPBYIMw2j2rF8Pw4bBsmXw6ZMrGP39ePXTu3fX50cegfT0ermXiaZhGM2azEwYOhQyMmDaNDhm7euQlKQPj6fy9ZR6mdhi7rlhGM2XVatgxAjYsgW+/BKOOAKYmKEWZiAJCaqq6ekqnhnqtsdCTF3vaaJpGEbjUk3Igt1/XL5cBTM3F6ZPh0MO8X2RkqIueVJAH46cHIiMVDc9KWm7294FOtV1ueaeG4bReKSnq5DVcf/xjz/UJc/Ph6+/DhBMUNHNztaH11v5WmQHt70cKuq6ZBNNwzAajylT6rz/uHChCmZpKcyaBQceWO2A1FS46Sa9TmamPt90E5SUqJsegHc3RNPcc8MwGo+MWvYfayA9XdOKwsJUMPfbbyfXTU3d0cWvwW337EYZpVmahmE0Hikput8YSE6Ofl6NuXNh+HDdmpw9uxbB3Bk1uO27U3tuomkYRuOxs/3HtLQqh/30ExxzDMTFwZw50LfvbtyrBrd9d2rPG2zcxZ5g4y4MowWzi+j5t99q843kZG3x1rNn/d1aRH5xzg2pyzm2p2kYRuNS0/4jQHo6s/41lxMnnUW3hHy+fm4T3Xvu2/Drq4a554ZhND3S0/nqxmkcP+kceibmMuuUx+n+xj/rrRRyTzDRNAyjyTHtwXROnHkDfdplM/PCiXTpHlavpZB7gommYRhNiqlT4ZS3z2S/5E3MvOA1OrYp0C9qSUVqSEw0DcNoMkyerE2DB3XZyNcnPUH72IAewTtJRWpoTDQNw2gSvPWWjqU45BCY/u42korW7TIVqTGw6LlhGI3OxIlw8cXwpz/BJ59AXNwAiLupairSJZfUWyPhPcFE0zCMRuWll+DyyzV5/aOPIDbW98XOUpEaGXPPDcNoNJ5+Gi67TAehTZ0aIJhNGBNNwzAahcceg6uvhpNOgg8/hJg6twNuHEw0DcNocB54AG64QSPl770HUVGNvaLgMdE0DKNBueceuP12OPtsePtt7VrUnLBAkGEYDYJzMG4cTJgA558Pr7yifTF3ym6OwQg1ZmkahhFynINbblHBvPRSePXVIARzN8ZgNAQhE00RiRaRn0Rkvoj8LiJ3+z7fS0R+FJFlIvKOiDQz49wwjLrgHFx3nWreVVfB88/rZIta2Y0xGA1FKC3NEmCEc+4AYBAwWkQOAx4EHnPO9QGygUtCuAbDMBoRr1eF8okn4Prr4amnghBMUJe82jyfFl977pR839sI38MBI4DJvs8nAqeEag2GYTQeFRWag/ncc3DrrfDoozoQMijqMAajoQlpIEhEwoBfgD7A08ByYJtzrtx3SCbQLZRrMAyjnqhDYKa8HC66CN58E+66C8aP34VgVr/2gAGa7Q5qYebk6L7mJY3vmIY0EOScq3DODQK6A4cA/YM9V0QuF5GfReTnTZs2hWyNhmEEQR0CM2VlcO65Kpj33Qd33x2EYFa/9tSpmvVefQxvE4ieN0jKkXNum4jMBA4HEkUk3GdtdgfW7uScF4AXQGcENcQ6DcOgZosyMDADlc9TplQRstJS7VT0wQfw8MOqc7tkZ9desEBN1CZGKKPnySKS6HsdA4wCFgEzgdN8h10AfBSqNRiGUUd2ZlHOm7fLwExxserrBx/A448HKZjQpIM+NRFK97wLMFNE0oH/AdOdc58AtwI3iMgyoD3wcgjXYBhGXdhZqs+2bbUGZoqK4OST4dNP4dln4Zpr6nDPJhz0qYmQuefOuXRgcA2fr0D3Nw3DaGpkZKiFGUhCAiQmqtXpfx8QmCkogLFjYdYsePll7YtZJ9LS1Jqt4dpNEasIMgyjkp1ZfYMGqb9dLTCTt1cqY8bA7Nnw+uu7IZige6I1XLspBH1qwmrPDcOopDarL7ApcHo6OW99yuhXYvnf5r1564E1nPl/PXf/vk204XBNmKVpGEYlwVh96elsnfAsI189l1+29OK941/lzAXjmkRdeENglqZhGFXZhdW3+c3PGTXjThZmd2bKme9yYt81kJ20Q/pRS8VE0zCMoNm4EY558RyW53di6ln/4bg+y/WLJpwiVN+YaBqGERTr1unws4yCjnw69nlG9Nlc+WUTThGqb2xP0zCMXbJmDQw9vITMFSV8fuh4Rix/EZYsaXIzyRsCszQNw6iVVatgxFElbMmq4MtTnufw/uGwfICWORYWajpSE5lJ3hCYaBqGsVOWLYMRIyBvi5ev0p7l4P55gAf22Qc6dNDoehOsDw8lJpqG0RoJos3b4sW6h1lSAjOPfYBBfR1VdvRaUfAnENvTNIzWRhBt3hYsgGHDtC/mrFkwaLA0q/rwUGKiaRitifR07abx888wf76ak/Pn6/trroH0dObPh+HDtV/H7NnaD5i0tMqATysM/gRiomkYrQW/hZmVpfuRW7bA9On63KEDZGXxy+2TGX50OdHRKpj9/W3Dm1l9eCixPU3DaC3427517Ki93PLyICpKn2Nj+SF6GKO/uo3EmHxmzklkr72qnd+M6sNDiVmahtFa8Df77d9fOwYXFEBkJBQU8M2WfRm14F90aFPEnGPv21Ewje2YpWkYLYFghp6lpOg+ZOfOcPjh8PnnkJvLLIZxwspn6RGxnq97/o1u7brXfA8DMEvTMJo//r3KpUth+XJ491047zyYPLnqcYHBnI4d4cgjmS7Hcvy2SfSKXM+sfa+iW/FyWLu21XQs2h1MNA2juTNlig4ZX7BA3e7kZB3/eO+9VcWvWjDnM45n7LbX2Scyg1nxY+m8/ldo0wbi4/WaRo2Ye24YzZ2MDI1oR0dDTIx+lpAAmzbt2K7NF8z58EM44wwYGL+cL7teSvv4OIjuoKLrL480asRE0zCaOykp8MMPamH68VucGRk77He+F3cR59zek4MOgs/DryUxPw9iEvW8mBgtAdq2rXF+SzPA3HPDaO6kpUFEhFboOKfpRMXF0K2bphOdd57ucy5fzqQvkznrlh4cNrCAL7+ExA7hmqxeVFR5rterg9SMGjHRNIzmTmoqnH8+rF+v1T0bN0JcnFb5fPmlvo+J4bU1x3De91dydNJvTGtzGm2vu1gtyh491MLMzdXngQO1c5FRIyaahtHcSU+HmTM1Ih4drUL4229qOUZEQFgYLywdzkUr72Jkmx/4NOxk4ras1rrzrl1h0SK1SseOhQMOgLCwVlkeGSy2p2kYzZ1nntFUo7ZtoV8/7edWWAhbt0JcHE9tPZu/5T/A8WGf877nfKKLi6FjLy0u79tXr7F2rVYHpaS0qt6Yu4OJpmE0RwKDO599prXj/sh5RcV2d/tfciM35t/ByWEf807UBUSVFqoF2rFj5bX69FEL9ZVXGue3NDNMNA2jueFPZk9KUhe7okL3M71etTDz86G8nH+W38wdBXdwuud9JkVcRIQrh/Dw7c052G8/vV4rbfG2u9iepmE0F9LTtUv6BRfAH39oapDHAz17QlmZzqUoK8OFhXN3yW3cUXEv54S/y1tRFxHhLYFevWDMGHXjs7JafYu33cUsTcNoDvity4oKWL1aU4Pmz4e994b994eVK6GkBJe9jTuLx/FPbuPCyLd4KfwKwhLjoV1PaN9ej42I0NGSmZm2h7kbmGgaRmNTW7MN/3cffqiCmZOjOZgej77/4w8NApWX4xzcJI/yL3c1l0dP5Nn4W/HEttPqoKgojapnZ6uL/sQTJpS7ScjccxHpISIzRWShiPwuItf6Ph8vImtFZJ7vcXyo1mAYTZ7aRk8EfldYqO53Zqa65UVFUFqqLnZZGc7r5Rr3OP8quZqro1/iuaQ78Pj/6z7iCK1Fh1bdPLi+CKWlWQ7c6JybKyLxwC8iMt333WPOuUdCeG/DaB74GwMnJel7/7O/YUZSkopkbq5amM5Vnut77XVwJc/zApdzgzzGI+4OJC9cB/wUFsLcuRotnzBhR7EMpqWcUYWQWZrOufXOubm+13nAIqBbqO5nGM0Sf2PgQPxTHv3fLV5c2bmoGhWEcQkv8wKXczv/5BFuQspK1QoFDRBlZu44FA2CGrBm7EiDRM9FpBcwGPjR99HVIpIuIq+ISFJDrMEwmiQpKTuf8uj/LicH2rWD2FgVTufAOcoJ43wm8hoXMT78XiZ4xiFhvv+kvV7dx0xMVAFduBDuu6/qfQKtXI+n8rW1hauVkIumiMQB7wPXOedygWeB3sAgYD3w6E7Ou1xEfhaRnzdt2hTqZRpG41DblEf/d5GRuocZGalBnOhoyojgHN7iLc7lfu7gH9EPIRHhWgIJKqxhYSqGsbH6+quvqlqRtVm5xk4JqWiKSAQqmJOcc1MAnHMbnXMVzjkv8CJwSE3nOudecM4Ncc4NSQ5seWUYLYnapjz6vxs8WEsinYPYWEriO3B65Ee8xxk86rmZ2z0PaiQ9MRH22Wd7vTnR0fpaRMU2LKyqFVmblWvslJAFgkREgJeBRc65fwV83sU5t9739lRgQajWYBjNgl1NeezcWYeP//ADxWFt+POW5/ms/DieTLqLqzt9AiU9KwUwIUGt0sxMPdc5DQiVlGgSfKAVmZame5ig5+XkqGV7ySWh+60tgFBGz48EzgN+E5F5vs/uAM4WkUGAA1YBfwnhGgyjeREYzY6KgjVroHdvSE2lcOlaTvnjQaZXjOD5fR7h8qQvIKtYrckDDqgUvvh43ccsLVULNCxM90T79atqRfot2cDouSW675KQiaZz7htgx3AffBaqexpGo7MnKTzVa8q/+EJTjbp3J39tDmOX/ZvZFYfzStSVXBT/E6D7mwwYoOf47zlhAixZojOCyso08t6tm7ro1cslbZZ5nbGKIMOoL6qLnj+FJ9hk8uo5m6WlEB9P7o+LOH7Jv/m+NJU3Ii/l3LKJ8HuEllAefbQK4/jxVa+Vmqpt3ywHs94x0TSM+qK2RPVgxCojQ8XWT0IC23I9jP79YX4uG8jbURdyuud9iI5TC3PLFh1n4e+J6ae6tXvddSaW9Yh1OTKM+mJPU3iqRbO37nUQIxc9wdyygUxufwWnt/msMv8yKkotzN9/r+pyW8J6yDHRNIz6Yk9TeAJyNjflRTP8kxtYUNqXD/rczCll7+ncn969Ne+yqEhbvO21V1Ur0hLWQ46JpmHUF7UlqgeSng5XXKHDywYPhiuv1M980ewNET0Y9sp5LNnWkalPZ3LC+5doL8ykJG0g3KmTzvY54ogdB6BZwnrIsT1Nw6gvgknhSU+HO+7Qdm7x8frZrFmaVzlhAmvbpzLi41QyS+GzL2H48L31mHHjNBq+aZNGw3v33jEanp4OK1boDPSOHaF/f83xtIT1ekVcYNeUJsqQIUPczz//3NjLMIy6Uz0os2ED/Pqrfuef6bN5M2Rnk5GYyojMiWwsjGdar6s4qs2vcNhhcNVVKrzVrzVgACxYoO8jI3U4Wtu2OonS41Frd8AAFVdrB1cjIvKLc25IXc4xS9MwQkVNKUiffqrt2iIiNAIeGwtbtrCyrDsjsl4guyKa6TEncljURoiMhdmz1Qq9//6qOZWTJ1fNw9y6VSt/hg2DI4/UsbxZWdqh3RoO1ysmmoYRKqqnIPmbBxcXV0a/MzJYGr4vI/I+ooBYvm5/Bgd550FmhYpqWJiOtwhMW0pPV8EUUcEsLlYr0z/DfNgw3ff0elVwTTDrFQsEGUZ94x+ANmkSzJsHGzfq54sX6x5jWNj2fpeLy3ozNGcqxRLNzB4XcJD3fyqCpaUqrM6pIM6bV3n9KVNUcBMSVDhjYvSRnV01em97mSHBRNMw6pPAPMmuXVW4vvtOhTMnR93xvn2hVy8WlOzD0NLpeMXDrH5XcEDxjzp+t6JC9yhF9BEZqfN9/GRkVFqYfjp31nMjI23KZIgx0TSM+iTQJd93X7UURbQJcFmZRrfz85nHIIYVfEJYVBiz4sayf8QSFVmvt2rX9ZISjbInJlbeIyVF90iLi9Xdd06FtlMnOPDAHVvMGfWK7WkaRn0SWArZuTMcfrjuM65YoS50fDw/hx3KsfMeJU5ymdHtfPrs10GDQzk5modZWKhBnYoKzc/s10/7ZPrxt3QbMEAFctMmDSzdfz+cdlqj/OzWhImmYdQnKSnqFq9bB//7n9aGR0fr/uOoUXyf0Y3RH19Fu7AcZu77V3oVZ8KQM9g+OnLjRvj2WxXN006r7HEZ6GYH5oNGRcHw4daMowEx0TSM+mTAALjxRli/Xq2/mBi1HLdt47/z23L899fSOSGfGee/Q4/4VPh4tQqjP8LeqRMMHKjBn8zMnfe4tJZujYaJpmHUF+np8Npr6i779ya9XthrL2Zs2Jexc26nR7s8Zlwwka7xeZCdo8nr2dl6vr+JcFiY5VY2YYISTRHpCezjnPtKRGKAcN9YXsMw/DzzjJZHVlRolBygvJwvNhzAKdtepbes4OtT36JTG48KZna2utlg3dObEbsUTRG5DLgcaIdOkewOPAccE9qlGUYzIj0dPvxQBVNEI99RUXxSMYY/Z7/GvmFLmN7+bJLDjoLM0h3F0USy2RCMpflXdGLkjwDOuaUi0jGkqzKM5oQ/N7OiQvcxY2MhJ4cPyk/kzPJJHMB8vog9nXZD9tf9TUsFatYEk6dZ4pwr9b8RkXB0KJphGKCutV8wt2yBsjLeiTyP08v/w0H8wldtTqHd6EPg4IOtt2ULIBhLc7aI3AHEiMgo4Crg49AuyzCaOIEdh777TkWzXTsoKeHN/FO4oPQFjpDv+WzvvxE/cqzmbIL1tmwBBCOatwGXAL+h43Y/A14K5aIMo0nz6KPw0ENarZOYqCWOItC7N6/EX8ulW//OsPBv+DjyNNrEdIIff1Sx7N9f8yqtHrxZs0vRdM55gRd9D8No3UyeDPfdpz0q27bVMsbCQvB4eG7BkVxZchfHRs3ig/aXEVvh0SR1f67m7NnaPPj++xv7Vxh7QDDR85XUsIfpnNs7JCsyjKbMU0+pVRkbq3mYJSVQUcETZVdwLY9zQuR0Jrf7C9EFW6FHDzjqKC2jzMlRke3Rw4JAzZxg3PPArsbRwOlo+pFhtB78e5hz56po5uSoFenx8Ii7kZt5iFM9H/H2fvcTGZkMf2zVjkOdOukDKvtbGs2aYNzzLdU++reI/ALcFZolGUYDUX18xM7qtwM7sLdpo13SfZ2IJsjf+bu7lzN4lzfbXUfEeq/uXfbura57INbfskUQjHt+YMBbD2p5Wvml0bypPopi6VI47zwdiTtoUFUB9bd7W79e9ybLynDAeMZzj/sH/8cbvBp9FeEpfdWaHDYMlizRmeTZ2ZXlkdnZmtBuNGuCEb9HA16XA6uAM0KyGsNoKAL7Xm7cqAPKRCqb9z7ySGUSekaGdiv67DPwenHi4XZ3Hw9yGxfJa7wYfwNhEVGwapX2vnz3Xc3ZPP98Pc/KI1sUwbjnwxtiIYbRoAT2vVy0SNu3RUdDbq4Gd/74Ay64AA49VEfsrl6tghkWzo3uER7jOq7gOZ5ucyue6GgVy4wMDfYkJ0O3bjB/vlX/tEB2KpoickNtJzrn/lX/yzGMBsLf9zIpqTKyXVysHYa+/17zKQsK4IsvNHjjHF48XFPxGE9zNdfwBP+OvAWp8IDE6P5l375VmwBnZ1cdiGa0CGoro4zfxaNWRKSHiMwUkYUi8ruIXOv7vJ2ITBeRpb7npD3/GYZRR9LSKl3xtm1VOP0zd6Kj1VUvLdVHeDheTzh/4Xme5mpu4mH+zbWIt0ItzBEjNBfzyCOr3sOqf1okO7U0nXN37+G1y4EbnXNzRSQe+EVEpgMXAl875x4QkdvQiqNb9/BehlE3ArufJyVpVU/XrvDrr1oS6fGoEJaXU+GJ4JKKZ5nIhdzJfdzLOAR0Nk+PHmphJiRoipGfDRv0WqWlOpnSOqu3GIKJnkejZZT7o3maADjnLq7tPOfcemC973WeiCwCugEnA8N8h00EZmGiaTQGgd3PJ0/WWeL+xhvt20NWFuUlFZxf9Dz/4WzukX8wjnu11HrE6aoAACAASURBVCM6Wi3Mgw9WUfRH40Et1jlz9PXRR+8YWDKaNcF0OXoD6AwcB8xG+2nWqQGxiPQCBqPt5Tr5BBVgA9CpLtcyjJCwYIGmCp12mgZyYmMpa9+Zs/Oe5z/ubP4Zdifjwu5XCzQiAuLidCTFRx+pYPot16QkrTVv21av16VLZZTeuhu1CIJJOerjnDtdRE52zk0UkbeA/wZ7AxGJA94HrnPO5YrI9u+cc05EamwzJyKXo82PSbGEYCPU+KPpHg8ccQQlvy/jjMUPMNU7nH+1+TvXlzykx8XFVQ5Bi4zUAFD1/M7Aa/mx/c0WQzCWZpnveZuIDAASgKCaEItIBCqYk5xz/v/NbhSRLr7vuwBZNZ3rnHvBOTfEOTckOTk5mNsZxu6TkqLBIKCoXTdOXf8MU3OH81TsLVzf5kUN+EREVAaLIiN1H7SkZMf8zqio7dfajlUDtRiCEc0XfBHuccBUYCHw4K5OEjUpXwYWVUtPmgpc4Ht9AfBRnVZsGKHAF00vXLiKk54ZzefLevNC7HX8NfZV3eeMidESytJSTUXq0kVd8KQktSJzcyvdcOcqRdTrrXwdOIbXaLYEI5qvOueynXOznXN7O+c6OueeD+K8I4HzgBEiMs/3OB54ABglIkuBkb73htG4pKaSP+pUjv/sr8zYNphX+z7AZZETVfQ6dlQrMyxMuxvFxsKJJ6qYRker9ZmQoNdJSFBh9e9vZmbqswWBWgzB7GmuFJHPgXeAGc65oEZdOOe+AWQnX9tQNqPpkJ5O7n8+5fgnRvND4UDeOOQpzqmYAst98c6SEm3CERcHmzdrACgwv9M5ONDXosHvhttc8hZLMJZmf+ArdMDaKhF5SkSOCu2yDKOBSE8ne8IzjHr1bH4sHMDb3W/mnEXjYNkytS5Bgz0rVuj8n4oKGDmyqis+YIBG3M0NbxUEU3teCLwLvOvb23wcTT0KC/HaDKNuBNvqLYAtb05j1Mw7WbClK5P3v4uTs99VYXRORXHLFnXRi4u1JdzgwTBuXOV1q9/TmnK0eIJq8SYiQ4EzgdHAz1iXI6OpUb3VWxAJ5VlZMPKlc1iS25mPzvoPY+K3wLtFKpKgUfG4OI2Ul5SoNXn//VWvZ254qyOYiqBVwK+otXmzc64g1IsyjDoT2OoNKp+rN8xIT4dnnmH9f5dxzPIXWFXalU+GPczIfUqBzrD33tr1yDl1z7t00QAQwJgxJpBGUJZmqnMuN+QrMYw9IbDVm5/qCeXp6XDHHaxdnMeIta+ztiyZaZ0uZOiSb6Dz0dCnj9aRb9iggunPD87N1U7stldpENyepgmm0fQJbPXmp3pC+ZQprM4MY0TmJDaVJ/FF6i0cGbkSCpNg3TpNH+rbV8Xx66/hhx/U4hw6FK66yqxMA7CxFUZLIS2tasOMefM0iDNq1Pba8BW/FzF84bPkeOOZnnoTh7ZdBC5a9yv33hteeaXyeoF9MQ0jABNNo2Xgb5jxzDNqJbZvD8cco671eeexNPkIhv/3Hoq8Eczo/1cObOtz24uLtewx0CLdjSi80Xqwzu1GyyE1FTp3hhNOqDL7Z1HJ3oz4732UE87MyONIXbUANsboOaWlOmJ3wAB9H0wU3kS1VVObpenvzt4POBitGQcYC/wUykUZRlDUJF7VZv/85t2fY1Y8jcdbzqzeF7E/2bA5SgeegXYnGjwYpk7V/cxdReF3I7XJaFnssnO7iMwBDnTO5fnejwc+bZDVGcbOSE+HO+/UZMuSEh2X+8svKmQ5OZCUxK/rOzNqxTNEUcKMmBPot3W11o2LwAEH6HViYlQs/fN8dhWFDza1yWixBLOn2QkoDXhfijUONhqSmizKZ5/VUse2bVXUiov1fXQ0xMTw04YUjlvxIm3JZUbMifT2rIRY33EFBVBeronr/hZufmHcVRQ+mNQmo0UTjGi+DvwkIh/43p+CjqkwjNAT6A5HRMC0afDmmypsnTurpQj67BwsW8Z317/H6Mt60CEym5kxJ9GzXQGU+ATTX+WzYYOKn787kV8YA6PwCQn6eXa2lkdCcKlNRotmlw07nHMTgIuAbN/jIufc/aFemGEAamFWVMD06TBxIvz2m4pUQYEGevLzqxw+J28wx17Vh849o5jzR2d6HtVDq3r84tqhgw5DKyjQyHq/flUbbQSOraiprVvgFEvrldkqkWA6vfm6Gu3jnHtVRJKBOOfcypCvzseQIUPczz//3FC3M5oSp5yi+5Vr16qV6PGoa11erkIYH6+VPMXFfL1+P8aue46eSbnMOOZ+uvRPUIsyKkrFb+FC+OknFbk2bXTkbnx83SPgFj1vMYjIL865IXU5J5ja838AQ9Ao+qtABPAm2mTYMOqX6oK0ZIlW65SWag14RIQKZ2QkhIfr5zk5fF40lFPXPkSfNuv56pSX6NQlXsVx7Vp12xMSYPFiSEzUfdABA/T8666ru+BZk45WTTD9NE8FTgIKAJxz66hMRzKM+sO/f5mdrfuNS5bA6tVQVqZWptcLhYW6N1lermLYsycfd7iIk5c/Sv+kLGae9QKduoWrsCYlaaVPjx4qnuXlKppHHKERc5sQaewGwQSCSgOnRopImxCvyWitVE/nWbdOgzb+nEr/UDMRtTrDwphCGmfOvobBBwpf7PMgSZ3b6TEbN2q3om3b9PiePXUGuU2INPaQYCzNd0XkeSBRRC5Du7i/FNplGa2SjIzKaDbA+vUqeGVlGgwSUeuyogIiIni7z985Y+6tHNw5k+nTIWmfDhokWrhQBfj332HTJrUwV66E5cur3s+i3sZuEEz0/BFgMjqKtx9wl3PuiVAvzGiF+MfobtgAn3yiIpqTU5mQ7vXqc2wsr8f8hXN/uoYje6zhi2H/VK1NS9OxFDNm6PX8zYNLSqBrV1iwwKLexh4TTCDoQefcrcD0Gj4zjPojLQ3uuEMtwtxcjY4XFKjoxcdr4KaigpeTb+Oy1XcyPPFXpp7wNm06dtTzU1OhWzcVVpGqTYQLC7VkMinJRlMYe0Qwe5qjgOoCOaaGzwxjz0hN1aDNpk1aHhkfrwnsq1ap6MXH82zOOVy1ehyjk35gSscrickfCFfcVHmNDRvUMi0srPwsOlqvOXw4jB/f0L/KaGHU1uXoSuAqoLeIpAd8FQ98F+qFGa2UkhI47ji1MouK9LmwEAoKeDzqFq4rvoGx7b/lvQ5XEdU5acfuQytXVs4iLy6GNWs0Yh4dba64US/UZmm+BUwD/gncFvB5nnNua0hXZbQs6pIM7i9T7N9f9yYzM6GggIcK/sqtm28grddc/nPC+0TmHqCCCWo9ZmTofmbXrhp179hRXfy8PN0XnTDBXHGjXthpIMg5l+OcW4WO7N3qnFvtnFsNlIvIoQ21QKOZUz330t9KLT295uP9ZYq5vikr5eXcW3Izt5ZP4Ky4T3h74AQiO7StFMzAa2dlaT5m377ahLhdOxg4EA47zDqxG/VGMHuazwIHBrzPr+Ezw6iZurZS89d+X3MNLiycu6If5r6s8zkvdT6vHv0VYe0HVu5Ljh+v1yspgTlzdC45aNT8xBP1dfXmGoaxhwQjmuICCtSdc14RsTEZRnDsTiu11FTcXntzW+m9PLT4T1wyeC7Pn/gxYbStel5GhkbIf/hB9yy7ddP2cAsWqKWalKRu+oQJofltRqskmOT2FSJyjYhE+B7XAitCvTCjheDPvQwkJ0ebaFxxBQwapJ3Tr7xyu8vuHFy/6HIe+v5PXDnkf7ww9mPCPG7HZPSUFB2g5uuhuT3VKCys0r0PoiGNYdSFYETzCuAIYC2QCRwKXB7KRRktiJpaqa1Yodbg7NnqSkdEwKxZcOedeOel89e/wuM/Hsa1/b/g6cPfxOMqak5GT0vTiZPO6WP9es3l7N9f8zPHjNF55VZfbtQjwcw9zwLOaoC1GC2FwGh5VJSmDv32mwrbYYepG52Vpd2GYmK0J2ZuLhXf/8Rfxqzh5Q2p3HJhFg9EfYj894fK86rP4UlN1RG9c+eqZVlRoTXm4eGV/TOtvtyoZ2rL07zFOfeQiDwJ7ODjOOeuqe3CIvIKcCKQ5Zwb4PtsPHAZsMl32B3Ouc92c+1GU2TyZLj3Xq0Xj4nR4Ex0tDbLiI6ujIyXlKig5edDZiYVngguKniSN4pPYFy7p7h7+oNIt65wyCGV51UnPV0FdcsWjZZ366bCWV6uLj9YfblR79Tmni/yPf8M/FLDY1e8Boyu4fPHnHODfA8TzJZEeroKpggkJ2t1Tl6eWn5//FEZRd+2TS3Q4mLYvJkyTxT/t+1J3ig+g3vaPMg94fcixUWao/nee/DttyqEgW62P5UpKkrnm4Nar4WFsP/+GgCy+nIjBNQ2jfJj3/NuzQNyzs0RkV67tyyjWTJlilqYyckqnBUVKmq5ufr5rFkqmMXFeszmzZTml3J2wYtMKT6BB2PGc8te78O6cq05j4pSK3P9em02HFgaWT2VqUsXFUj/HHOrLzdCRG3u+cfU4Jb7cc6dtJv3vFpEzkct2BudczX4XSAil+MLOKWYe9U8yMhQMdyyRd3u/HwVT9B68IQEDfxERUFiIiXJ3Tl9xYV8XHocj3V/lOvavA3tOmmtub9Lu3OV84CysjQ3s/p8cz8JCWqdWn25EUJqc88fAR4FVgJFwIu+Rz6wvJbzauNZoDcwCFjvu36NOOdecM4Ncc4NSU5O3s3bGQ1KSoq60cuWqWvunO5dFhbqTB7Q94MHU5TSj5PT7+Xj0uN45pDXuC4tQy3EnBxtFBwVpdZpUZGeX16uouivKIqKqjmVyf4Ha4SY2tzz2QAi8mi1wUMfi8huTTlzzm30vxaRF4FPduc6RgMTbO34gAHw+OPqUjunrjKowBUWamBo8GAK2vXgpP+czcxVvXjpJbjk4ANhyqrK/c6UFD0/O1v3ROPjVVDbt690x0tKKoNDNY3aNYwQEUyeZhsR2dv/RkT2AnZr5IWIdAl4eyqwYHeuYzQgdakdX7BA671jYlQwKyr087IydbGBvNIojn/rXGat7sXEUz5UjUtNVZf6ww/hjTd0SmRxsd4vJQV69VLrs39/vV5Cgl6/tlG7hhEigimHvB6YJSIrAAF6An/Z1Uki8h9gGNBBRDKBfwDDRGQQule6KpjrGI1MXWrHMzK0DduWLbqXGR6uglleDiUl5GTmMeZ/F/FTUQ8mHfksZ939px3vl5oKzz1Xad1+9JFe6/DDtbcmVLrhNhXSaASCSW7/XET2AXz/m2exc64kiPPOruHjl+u4PqOxCaZ23C9wc+dql6GiInWv/RMjw8LIjunKcWte5tfS/Xlnv7v589On1S54fkFMS6vcw/R6zQ03Gp1duuciEgvcDFztnJsPpIjIiSFfmdE02FntuD/gEui+H3qouuGlpWph+uq+N0syIwqmMr9sP6ac+S5/PjQzeAvR3/XI3HCjiRCMe/4qmsx+uO/9WuA9LIjTOvBberBjwCU9Ha65RlOBOnbUdKPy8iqnZ5HMyPIvWEJfPupzI6M7eSCpjhFuc8ONJkQwotnbOXemiJwN4JwrFPEn3xktHr+lFxg997vGjzyigtmhg7rkM2aoC+3xgNfLejpzDF+zil586jmJY8qXwvL94f77a75XXTq8G0YjEYxolopIDL5EdxHpDexyT9NoQdRk6fkbAMfG6vTIigpNF3IORMikGyOYwTq6Mo0xDE1cAB376OC0moTQ7+YnJVWN0psrbjQxghHNfwCfAz1EZBJwJHBhKBdlNAP8DYD9zTd81iXAKteTEXzNFtrzZeRYjvD8BGdcqO57ZmbN16trh3fDaCRqFU0R8QBJQBpwGJpydK1zbnMDrM1oyqSkwPvvb8+/9Df9Xc7ejGAGubTlK0ZycPlcza/s1Emtx51V7OxOh3fDaARqjZ4757zALc65Lc65T51zn5hgGoBW/6xapYGf2FgA/gjfn6HMJp84ZoQfx8Fhv+qxXi+8+6427BgwoObr7SpKbxhNhGDc869E5CbgHaDA/6GN8W2B+AMx8+bp/mRioo6jqCkg8/XXWi6ZlwcFBSwMG8CIkk/w4mFW3FgGtsmAkngtn9y8GfbbT/tdTp2q0yKrX6+2KL1hNCHE7WKGioisrOFj55zbu4bPQ8KQIUPczz/vVrm7ESz+QEx5uZZD+vcoBw7UjkOBAZn0dBW56GjYtIn08v0Ymf0uYa6cr2UU+3XyNQXetEn3MSMi4OST9Vz/dMiaOhFZ9NxoYETkl2q9NXZJMBVBe+3+koxmgz8QM3++1o7HxGga0dq1cMABlQEZf25mTg4UFTE39ihGrX6RGFfIDM8o+kathuIodd2d01LKLgEtB2rbp7R8TKMZsEvRFJFo4CrgKDTt6L/Ac8654hCvzWgo0tO1xts5benWrZt+Hh2t4ugXOr81mpUFPXrw0/L2HJf3Mm0ln5nt0ti7dC0kJWtwKDJSXfPCQg0SbdigteO2T2k0c4LZ03wdyAOe9L0/B3gDOD1UizIaEL8QRkZurxNn9WrtLFRcrI2EJ01Sq/OjjzQfs6KCbzmSMQXvkSybmRExmp6RedCrr16zsFDPq6jQ6wIsWqT147ZPaTRzghHNAc65/QLezxSRhaFakNHA+N3ywYPh++/19YYNOmbXObUys7K001BREYgwq/woTnST6Sbr+Do+je5shIQucPDBWhXk8UBcnFqqbdvq+UVFMHy4jZ8wmj3BiOZcETnMOfcDgIgcio6qMFoC/vxIj0fbry1erA031q3TenKvV/c3IyKgqIivyo7mJKayFyv5yjOaLrlr9buKispEd38pZXKyimdYmEbibQyF0QIIRjQPAr4TEf/ufQrwh4j8hkbRzWxoLtQUnU5JUZe5pEQFMydHrU0ROPNM+PhjddPDw5nmPY5TmUxflvKV51g6uiwVzLg4GDJEo+4lJXpux4464qKoSEU0MbGxf71h1AvBiGZNY3iN5sbOartPOglee03rx+PjVQSzstRqfOMNddG9XqbmD+d07/Psz+9M51jae7eoBdm2rT737auW5fz5Gkhau7YyiNSnD+yzT2P/BQyjXggm5Wh1QyzECDE7q+1esECbaGzapG65fwpkhw7bE9wnbzyKs4tf5EDm8jljSPLkgPN1Zi8trRTEhAS1KMPCNE0pMEndZo8bLYRgLE2jJbCr2u7jjtN9yFmzVPSiowF4y/N/nF98O4fKT0yTE2jrciAiUl1w/0iLgw/Wa+TkVFYQVW8lZ8Efo4Vgotla8O9d+i1MqJoz6f8uJ0dd7i1bmLjlRC7acidHe77hE8YS1y0Bojqo6x4To6lF8fHqlmdnV6YTWZK60YIJZhql0RJIS6sUNq+38nVaWtXvPB5YsIAXF/+Ji7Y8zDEyk8/CTyYurEhzL3NzNcjTqxeMGqWuvY2hMFoRZmm2dAIj5kVFsHChWpPdusHVV1eWRsbGwrRpsHEjTxdcyNUVjzMm7AumeE8lWrzQJk73LyMi9PyKCrVSn3jChNJoVZhotmQCI+YRERr0ARgxQvcsp07V91On6jFduvBYweXckHsTJ0V9zrucSZSrAE+YuuEFBeqWb9sGe+9tlqXRKjH3vCUTGDH/4w/dq2zbVl/7P3/qqe2vH1j6Z25YdxN/jv6U99pdQVTnJD3eOc3VjIvT6HjPnmZhGq0WE82WSHq6Vt9MmqS9MTduVJc6OrqyCQdo9HztWkhI4J7ZQ7l9/TWc3e4L3t73biKLc9W6dG57k2FE9P24cSaYRqvF3PPmTvUqnwEDKt3trl1VIL/7Tt3zYl9jqoQEfc7JwXXtxrjPj2TC/4Zzft/veSXqNsK8DvbaS9OJYmJ0VEX37jtvSGwYrQgTzeZMTVU+996rYpmRoY03/FU54eFa6ZOXpw2Cp03DJXfkloTneWROPy7d/zueP/ULPMv3hd9/16i4iaRh7ICJZnOmpiqfvDyYO1dTguLiYOtWbfUWHq7vPR7YsAG3dh3XeS/liaJ+XNX7c54Mux3PJ8Bhh2n5pAmlYdSIiWZzpqYqn4oKKCvT58xM7WeZkKBiWlQEHTrgxcNVeQ/yfPEFXB/5NI+GP4UcdrTud2ZnN85vMYxmggWCmjM1TXD0146vX6/WJWjwpqICoqOp2JzNpfn/5vniC7gt4lEe9dyMJFSLqE+Z0vC/xTCaCSETTRF5RUSyRGRBwGftRGS6iCz1PSfVdg1jF9RU5RMfDwcdpCJZXKy5lWVl4BzlFcKF+U/xatFZ3JXwOPeHjUM8smNE3WaNG8ZOCaWl+Ro7tpW7DfjaObcP8LXvvbG7pKZqgnlSUmUp47hxlZMkc3NVNAsKKCsXzt3yOG96z+G+2Pu5u81DiKA9L4uLq0TUbYaPYeyckO1pOufmiEivah+fDAzzvZ4IzAJuDdUaWgXVm2Okp2s+ZXGxjuMFSongLN7mA07lYbmZm8oeh+J4jaKHh6u4DhpUtemGYRg10tB7mp2cc+t9rzcAnRr4/i2fKVO0iqesDKKiKPbEksYUPuBUHo+6hZuin9bgUF6eCqQ/QPTTT1pbbqWRhlErjRY9d845EXE7+15ELgcuB0gxd7HmURU1iVtGhrrqzlFUHsEp3sl8yXE8K1dxhec13euMjVULMyyscmLkvvuqK28YRq00tKW5UUS6APies3Z2oHPuBefcEOfckOTk5AZbYJPEn8SenV11VEV6+o7H5ufD999TUBLGCRUfMZ1RvMzFXMFz6rKHhekDNAAUG6tiuW6dRc4NIwgaWjSnAhf4Xl8AfNTA92+eBCaxezw7Tw2aPBmmTyevLIoxTGM2Q3md87mYVzXtyDl123Ny1CXPz9eAUXl5ZeWQRc4No1ZCmXL0H+B7oJ+IZIrIJcADwCgRWQqM9L03dkVGRmV0209NAnfXXWzLEY7lS77jCN7iHP6PSfqdiAquXzwjIirF0j/f3CLnhrFLQhk9P3snXx0Tqnu2WHY1qgIgPZ2ty7ZynPuC+RzAe5zOqXxY+X1CQuWIiuJitTB9kyapqNB6dYucG8YusTLK5kBamu5hQqVFuGKFdl+/+GJISWHzqnxGlX/BQvozhTRO5NOq1ygs1L1Mvzhu26Y5mhERGhTq29eacxhGEJhoNmUCI+axsZoSlJkJUVEqnFlZUFLCxnnrOea3f7OcnkzlJI7jyx2vFR+v52VlQZcu0K8fDBtWacGOH9/Qv84wmiVWe95UqR4xj4rSKPd11+n3mzYBsC56b4YteoaV5d35tMOFHBf29Y7XatNG9zQ7d1ZXfP16Fc3A4WqGYQSFWZpNlZravvk//+EHiI9njaQwIv0xNpS14/Oef+FPGz5U972kRPcqS0u1HVxEhAqn16uzffLzNYrepYvNJDeMOmKi2VTxt33buBEWLaqcR56UBM6xqqQLw5c8wdayeL4ceBOHR/wBWyO1j+bKlRr0qaioTDMaM0Y7sJs7bhh7hIlmUyUlBZYu1QmS0dEqmDk5sG0by7oezYivbifPteHr1BsYEjEfcvM0mHPkkXDIISq0GzbA5s0aMEpOttpyw6gHTDSbKmlpcN55uhcZHQ1btsCmTSyOGMgxC/9OiTeCmW1PZtC6RSqIvXvDhRdWzgc6+uiqUfbMTBVic8cNY48w0WxqBEbMi4o06r16NWzezAIZyMjCt3BemNX1bAZ0zoWidpq0fuGFcNppam0G1qhPmGAiaRj1iIlmU6L6oLSkpO3D0OZHHszInMlEuFJmRBxH/7INsNYD/ftr7uWCBSqa1VvFGYZRr5hoNiWefVbHTpSWaiJ6YSGsW8cv5Qcwyk2hjRQxI3wU+8gyKIzQyHhRkQpmYWFjr94wWgWWp9lUSE+H6dM12u3xwKpVkJXFD+FHcYz3S9q6HOa0P5V9In2TJb1e3euMidHjt21r7F9gGK0CszSbClOmaCd10KBPVBTflBzMmMI36OTZxIyI0aTkZ2qTjfJyDRCVlGh03OOBxMTGXb9htBLM0mwqZGToyAnfMLSZZUdx3Na36CbrmB09mpTYzZVTJUFd8/JyFc3u3fVcwzBCjolmUyElRd3tww/nS89ojt/0Gr3C1jCrw+l02ytS3fCwMLVG27ZV0dxvPy2NXLvWSiENo4Ew97yx8acYzZsHCxfyWckxpG16iX6epXwVczLJJZthaxsVyfh4rerJz1ehzM3VssmkJIuYG0YDYaLZmEyeDPfeq2WOMTF8uPkozsh+nIERf/Bl5Im0L9mgx5WXa/CnY0d138PDNdUosEuRYRgNgolmqNjVILT0dBXM4mIoLua95YM5p/R5Dor6nc8TzyKxvADaJGrAp7xcp0a2aVN5vo3cNYxGwfY0Q0Ewg9CmTNk+RndS7ljOKn2dwzw/8aVnNInZK7UVXHS0Hturl/bT3LgRhg5VC7OsTC1MG7lrGA2KWZqhoLa2bn6By8gAr5fXis/i4rx/MzTsWz6OPp248m2Vc3zKy1U44+K0frysDJ57rnF+k2EYgIlmaMjI0B6Ws2ZVTnns16/qILSUFF7IP4e/5N3PqKjZfJh4EbHbtmnOZVSUVvqEhWl0vKhIrdJhwxrrFxmG4cNEMxTk58OMGbofGRurFuOcOTBwoPaxzMjgqaXH8rctZ3F8+Je8H3Y20blFKpbJyZUVP1FRmpcZHg59+sCVVzb2LzOMVo+JZn2Tnq7pQxUVKpiFhdtHU7BxI4SH8+im87npm9GcHDmNd6IvICrSAxGJKpL77qudiaD2QJJhGI2CiWZ9kp4O11yjnYliYnSmT0GBWo0eD4SH88+Zh3LHhtGcnjyTSV3GERHhG8ObmakW6vr1+t66FRlGk8Si5/WFP2KelaUVO6WlKoJhYdCmDc7ruLv4Fu7YcC3ndJzOW11vJiIhVvcqy8qgZ0+1MgsLd4y0G4bRZDDRrC/8EfOOHTXwU1y8PQruyiu4093H+LK/c6FnIq9nn0T4qmXaXLikpLJbUUmJnp+UpNczDKPJYe75nhCYwD53D+gQJAAAD/hJREFULhx6qFqL330HkZFQUoIrLeMmeYB/ueu5nBd41nslHn+S+tq1Wkvuj5YXF8OBB6roBkbaDcNoMpilubvUNJd8zhy1Lvv1g/JyvBVeruEJ/uWu52rP0zzHFXjCRNOREhP1PH97t5gYOOIIrS3PydHgj2EYTQ6zNHeX6gnsBx6oeZnffgsieCOiuNLzGC94L+XG8Md5uPx6xCN6/P776zlerz6GDNHPExKsNNIwmjhmae4uGRkqcn46ddIJkJs2UVHm5ZKip3jBeym3t32ahxPuQ6Ii1boMC6s8JzcX9t5bSyGTkjSCbqWRhtGkMUuzJnbVbAP08+odhvLzKXdhXLDhId4qPp7xyU9z177vIeynQZ/s7MrE9dxcffz975ZeZBjNiEaxNEVklYj8JiLzROTnxljDTgmm2QaokPpdaa8Xli6l7Lv/cU7xy7y17Xju7/Rv/lH2d2TLZg3ydOoEBxwAPXpoLmZiIjz0kE6QNAyj2dCYluZw59zmRrx/zRZlMM02QF/fdNP280vWZHFm2Vt8lDuER7s8wg093oOCbiqqxcUwciRcdZVZlIbRzGm97nn1GeN+izI3d0dhC0wBqkFoi/um8ud95vPZ2gN4ss/jXB32Emws1oh6x44wcaKJpWG0EBorEOSAL0XkFxG5vFFWEGhRejyVr7dt05SfQPwpQDW47oUPPMFJw/P4LPMAnu98F1e3e0td8U6d9LqHHWaCaRgtiMayNI9yzq0VkY7AdBFZ7JybE3iAT0wvB0gJRc5iRoYKXyAJCbrXuHy5NtkoKansPHThhZV15R07Qv/+5LdLYez7pzN7bRyvDJ3IRRvfA9pWvaZzNd8/mGCTYRhNjkaxNJ1za33PWcAHwCE1HPOCc26Ic25IcnJy/S8iJaVmi7JzZ23pFkheHrz2mgpmhw5QVETuN+mMfvUM5qzrzRtHv8hFe8/WlKOYGHXxY2L0fWnpjvcONthkGEaTo8EtTRFpA3icc3m+18cC9zT0OkhLU6ECtTBzclS82rTR3MmDDqo8dto0DeZ07AhFRWyLSGb0qgf4pbAXb4+eyOmHrAN8KUiBjYKzs6FLlx3vHWywyTCMJkdjWJqdgG9EZD7wE/Cpc+7zBl+FP/pdPam8pKRq0jroZyUlsO++bM2PZOSvDzO3sB/vdbue0zvMVAGunoLkf13TPPLqifFg9eaG0UxocEvTObcCOKCh71sjNSWVR0XBF1+oW52QoA04oqIA2BS3FyM33skfRR35oNvfOKH3YrjpicprBKQgkZKipZA1WY41JcZbvblhNAtab8pRTaSnw5o1uicZH6+9LWfNgo4d2RDRg2NeOY8VuR2YetKLHNu2sKpgQvCVPTvbGrB6c8No8ljteSBTpkDv3jBggAZ9li+H3FzWxvdn6LKXWZXXns9GPsaxg7L2rD58Z1sDtp9pGE0eszQD8U+RXLJE8yx79iRjW1tGfHEHGyPD+WJGGEcddUv93MvqzQ2jWWKWZiApKToUzddJfWVxF4Yufp7Nrh3Tz36Vo45q7AUahtHYmGgGkpYGW7aAcywt7MrR8x4np6INX5/0BId5v2vs1RmG0QQw9zyQ1FQYOZJF32VzzMLHKPOGMTNmDAf8uBqOPNKqeAzDMEuzOgtGXc+wxc/i9cKspDQOSFytUfSZM+Fvf7MqHsNo5ZhoBjBvHgz7Sz/CKWd2uzT2j1iigaHevbXj+rJlOzb4sKmRhtGqaN3ueXo6PPss/PADPxfsy7EZLxKXFMGMLmfTp2cYePpXPb6wsOp7q+IxjFZH6xXN9HS4805Ytozv5QhGr3icdp5sZu57F71ycyC3XDseBRIbW/W9VfEYRquj9brnU6ZAVhZzOJpjlzxJx8htzNnvSnoV/K4NO3Jztbem16vPYWHQp09wteWGYbRYWq9ozpvHjEVdGLP4X3T3rmF2/Fh6hK3TxhxxcTq/JzGxcp7P44/Dk09aFY9htHJap3uens4X/2vHKXlP01tW8nXMiXTK2wDLI3XwWUqKDjyraeiZiaRhtGpapaX5yQMLOGn9c/TzLGNm1Gg6SZY2Hi4uVkvTXG7DMHZCqxPNDz6AtHfOIDV6KTMGXENyUrl+IaINiPfbz6xJwzB2Sqtyz995B849Fw7uup7PO/6NhPBC6NtXvywq0udBgxpvgYZhNHlajaX55ptwzjlw+OHw5bs5JHRtoxHywkJ95ObqADVzzQ3DqIVWIZqvvALnnw9Dh8Lnn0P84QNgwgSd51NWpl3ahw6F++8319wwjFpp8e75c8/BlVfCscfqfub2/PTU1P9v7+5jq6rvOI6/PyJQ50McgxC2uSFsUdnCkFXj5sMe2BbmA0IkjBEXxGWIgcGIECHofEhM0DnwgclSnMKcQ4bM0DHnfBhomImCSMuDgqgsG2Ew42TgtIP2uz9+v8Kh3Nv2trbnd8v3lTQ999xzej78aL/9nXt6vjfcDeSccyXo0jPN++4LBfOyy2DlymNv6HHOuVJ12aJ5990wbRqMGhVu/qmoyDuRc64r6JJF8447YOZMGDMmXDHv0SPvRM65rqJLFU0zuOUWuOkmuPpqePTR0NnNOec+Kl3mQpAZzJ4Nd94JEybAokWhx4Zzzn2UusRM0wxuuCEUzEmT4MEHvWA65zpG2RfNhobwLhTz58PUqfDAA6GxunPOdYSyPj1vaIDrrgszyxkzQjc3Ke9UzrmurGznZPX1cO21oWDOmeMF0znXOcpypnnoULgtculSuP12uPnmvBM5544Xucw0JQ2XtE3SDkmzStn34EEYOzYUzLlzvWA65zpXpxdNSd2AXwDfBQYB35c0qDX71tWFZuorVsC8eXDjjR2Z1DnnjpXHTPN8YIeZvWVm/wMeA65saacPPgi3RFZXw4IFMH16h+d0zrlj5FE0PwX8PfP4H3FdUQ0NMGJEaOtWVQWTJ3doPuecKyrZC0GSJgITAXr2HMzBg/DwwzB+fM7BnHPHtTxmmruAMzKPPx3XHcXMqsys0swq6+q688gjXjCdc/mTmXXuAaUTge3AMEKxXAeMM7MtzezzL+BvQG/gnc7I2QYpZ4O086WcDTxfe6ScDeAsMzu1lB06/fTczA5JmgL8GegGPNRcwYz79AGQtN7MKjshZslSzgZp50s5G3i+9kg5G4R8pe6Ty2uaZvYk8GQex3bOufYo29sonXMuD+VWNKvyDtCMlLNB2vlSzgaerz1SzgZtyNfpF4Kcc66cldtM0znnclUWRbM9DT46g6SdkjZJ2tiWq3EdkOchSXslbc6s6yXpGUlvxM8fTyjbrZJ2xfHbKOnSnLKdIWm1pK2StkiaFtenMnbF8qUyfhWSXpZUE/PdFtefKeml+PO7TFKnv9VhM9kWS3o7M3ZDWvxiZpb0B+HPkt4EBgA9gBpgUN65mmTcCfTOO0cmzyXAUGBzZt1dwKy4PAu4M6FstwIzEhi3fsDQuHwq4e+JByU0dsXypTJ+Ak6Jy92Bl4ALgN8BY+P6XwLXJ5RtMTC6lK9VDjPNNjX4OJ6Z2QvAu01WXwksictLgJGdGioqki0JZrbbzDbE5f3Aa4S+CKmMXbF8SbDgQHzYPX4Y8E3g8bg+l/FrJlvJyqFoltzgIwcGPC3plXjPfIr6mtnuuPxPoG+eYQqYIqk2nr7ncvqbJak/cC5hRpLc2DXJB4mMn6RukjYCe4FnCGeJ75nZobhJbj+/TbOZWePY3RHHbr6kni19nXIomuXgIjMbSugROlnSJXkHao6Fc5SU/mxiITAQGALsBn6eZxhJpwArgJ+Y2X+yz6UwdgXyJTN+ZlZvZkMIPSXOB87OK0tTTbNJ+iIwm5DxPKAX0GKX3nIomq1q8JEnM9sVP+8FniB8s6Rmj6R+APHz3pzzHGZme+I3dAOwiBzHT1J3QkF61Mx+H1cnM3aF8qU0fo3M7D1gNfAV4PTYcwIS+PnNZBseX/IwM6sDHqYVY1cORXMd8Pl4Ba4HMBaozjnTYZJOlnRq4zLwHWBz83vlohpo7BM1HliZY5ajNBakaBQ5jZ8kAb8CXjOzeZmnkhi7YvkSGr8+kk6PyycB3ya87roaGB03y2X8imR7PfPLUITXWlseuzyvtpVw5etSwpXCN4E5eedpkm0A4Yp+DbAlhXzAUsJp2kHCa0g/BD4BPAe8ATwL9Eoo2yPAJqCWUKD65ZTtIsKpdy2wMX5cmtDYFcuXyvgNBl6NOTYDP43rBwAvAzuA5UDPhLL9JY7dZuA3xCvszX34HUHOOVeCcjg9d865ZHjRdM65EnjRdM65EnjRdM65EnjRdM65EnjRdMmK3XtmFFg/UtKgNny9/pLGZR5fI2lBe3MWOM4aScm+L45rHy+arl0yd3p0ppGE7j7HaCFPf2BcM8871yIvmq4oSTfHPqZrJS1tnPXFmdQ9sXfoNEnDJL2q0FP0ocamBwp9RnvH5UpJa+LyrXG7NZLekjQ1c8w5krZLWgucVSDTV4ERwM9i/8OBBfIsljQ6s09jd5u5wMVxv+lx3SclPaXQK/OuAscbLml55vHXJa2Kywslrc/2Zyyw/4HM8mhJi+NyH0krJK2LHxc2/7/hUpHLu1G69Ek6D7gK+BKhjdYG4JXMJj3MrFJSBeFOmWFmtl3Sr4HrgXtaOMTZwDcIfSG3SVpIuGtjLKHxxIkFjomZvSipGlhlZo/HrIfzxMeLixxzFqHv5OVxu2visc4F6mKO+80s21XrWaBK0slm9j7wPUJ7Qgh3f70rqRvwnKTBZlbbwr+70b3AfDNbK+kzhLe0PqeV+7oc+UzTFXMhsNLMPrTQu/EPTZ5fFj+fBbxtZtvj4yWERsMt+aOZ1ZnZO4QGGH2Bi4EnzOy/Frr3lNJjYFnLmxT0nJntM7MPga3AZ7NPWmhp9hRwRTz1v4wj906PkbSBcHveFyjykkER3wIWxFZl1cBpsXuRS5zPNF1bvd+KbQ5x5BdzRZPn6jLL9bT/ezGb5/BxJZ1A6PhfTGtyPAZMITRPXm9m+yWdCcwAzjOzf8fZbdN/IxzdRi77/AnABbFYuzLiM01XzF8Js6uKOAO6vMh224D+kj4XH/8AeD4u7wS+HJevasUxXwBGSjopdo66osh2+wmn9cVkjzuC8PJCa/Yr5nnCW3T8iCOn5qcRCvU+SX0JvVQL2SPpnFi8R2XWPw38uPGBWvPeNC4JXjRdQWa2jnDaWAv8idAJZl+B7T4EJgDLJW0CGgjvAwNwG3BvvEBT34pjbiCcZtfEY64rsuljwMx48WlggecXAV+TVEPo59g4C60F6hXeXGt6gf2K5aoHVhEK46q4roZwWv468FvCL5lCZsV9XiR0d2o0FahU6Bi+FZjU2jwuX97lyBUl6RQzOyDpY4RZ4MRY2Jw7bvlrmq45VfGPyCuAJV4wnfOZpnPOlcRf03TOuRJ40XTOuRJ40XTOuRJ40XTOuRJ40XTOuRJ40XTOuRL8H0fpoE6d+aXWAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 360x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQikz3IPiyPf"
      },
      "source": [
        "# **Testing**\n",
        "The predictions of your model on testing set will be stored at `pred.csv`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O8cTuQjQQOon",
        "outputId": "84a3e897-eff6-486d-f5da-70f77318274e"
      },
      "source": [
        "def save_pred(preds, file):\n",
        "    ''' Save predictions to specified file '''\n",
        "    print('Saving results to {}'.format(file))\n",
        "    with open(file, 'w') as fp:\n",
        "        writer = csv.writer(fp)\n",
        "        writer.writerow(['id', 'tested_positive'])\n",
        "        for i, p in enumerate(preds):\n",
        "            writer.writerow([i, p])\n",
        "\n",
        "train_loss = dev(tr_set, model, device)\n",
        "dev_loss = dev(dv_set, model, device)\n",
        "print(\"train_loss:\",train_loss,\"----dev_loss:\",dev_loss)\n",
        "preds = test(tt_set, model, device)  # predict COVID-19 cases with your model\n",
        "save_pred(preds, 'pred.csv')         # save prediction file to pred.csv"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_loss: 0.8897911951375106 ----dev_loss: 0.8928999017786097\n",
            "Saving results to pred.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfrVxqJanGpE"
      },
      "source": [
        "# **Hints**\n",
        "\n",
        "## **Simple Baseline**\n",
        "* Run sample code\n",
        "\n",
        "## **Medium Baseline**\n",
        "* Feature selection: 40 states + 2 `tested_positive` (`TODO` in dataset)\n",
        "\n",
        "## **Strong Baseline**\n",
        "* Feature selection (what other features are useful?)\n",
        "* DNN architecture (layers? dimension? activation function?)\n",
        "* Training (mini-batch? optimizer? learning rate?)\n",
        "* L2 regularization\n",
        "* There are some mistakes in the sample code, can you find them?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tmCwXgpot3t"
      },
      "source": [
        "# **Reference**\n",
        "This code is completely written by Heng-Jui Chang @ NTUEE.  \n",
        "Copying or reusing this code is required to specify the original author. \n",
        "\n",
        "E.g.  \n",
        "Source: Heng-Jui Chang @ NTUEE (https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.ipynb)\n"
      ]
    }
  ]
}